Sony Interactive Entertainment Inc.

Jian Zheng

Ruxin Chen

Sudha Krishnamurthy

Masanori Omote

Justice Adams

Arindam Jati

A system enhances existing audio-visual content with audio describing the setting of the visual content. A scene annotation module classifies scene elements from an image frame received from a host system and generates a caption describing the scene elements. A text to speech synthesis module may then convert the caption to synthesized speech data describing the scene elements within the image frame

Scene annotation using machine learning

For image captioning such as for computer game images or other images, bottom-up attention is combined with top-down attention to provide a multi-level residual attention-based image captioning model. A residual attention mechanism is first applied in the Faster R-CNN network to learn better feature representations for each region by taking spatial information into consideration. In the image captioning network, taking the extracted regional features as input, a second residual attention network is implemented to fuse the regional features attentionally for subsequent caption generation.

Growing community of inventors

San Mateo, CA, United States of America

Jian Zheng