Amazon Technologies, Inc.

Ahmet Emre Barut

Chengwei Su

Wael Hamza

Weitong Ruan

Melanie C B Gens

Matthew Cavell Johnson

Prashan Wanigasekara

Kechin Qin

Spurthideepika Sandiri

Techniques for performing an action with respect to displayed content are described. A natural language interpretation corresponding to a received spoken user input may be determined. Prior to receiving the spoken user input, content may be displayed to the user from which the spoken user input was received. The natural language interpretation may represent a request to perform an action with respect to a portion of the content currently being displayed. Content identifiers corresponding to content being displayed, may be determined, and embedding data representing at least one feature of the content may be determined using the content identifiers. The natural language interpretation and the embedding data may be processed to determine that the spoken user input relates to a first portion of the displayed content instead of a second portion of the displayed content. Based on the determination, an action responsive to the spoken user input may be performed.

Natural language interactions using visual understanding

Devices and techniques are generally described for selection of objects in image data using natural language input. In various examples, first image data representing at least a first object and first natural language data may be received. In some examples, first embedding data representing the first natural language data may be generated. Second embedding data representing the first image data may be generated. Relative location data indicating a location of the first object in the first image data relative to at least one other object may be generated. The first embedding data, the second embedding data, and the relative location data may be input into a multi-modal transformer model. The multi-modal transformer model may determine that the first natural language data relates to the first object.

Natural language selection of objects in image data

Growing community of inventors

Boston, MA, United States of America

Ahmet Emre Barut