Amazon Technologies, Inc.

Wael Hamza

Chengwei Su

Konstantine Arkoudas

Ahmet Emre Barut

Beiye Liu

Liwei Cai

Subendhu Rongali

Weitong Ruan

Rahul Gupta

Melanie C B Gens

Sachin Midha

Xinhong Zhang

Fabian Triefenbach

Md Humayun Arafat

Jonathan Jakob Hueser

Jin Cao

Mariusz Momotko

Chandana Satya Prakash

Jwala Dhamala

Jennifer Yuen

Dewan Muhammed Ibtesham

Devices and techniques are generally described for selection of objects in image data using natural language input. In various examples, first image data representing at least a first object and first natural language data may be received. In some examples, first embedding data representing the first natural language data may be generated. Second embedding data representing the first image data may be generated. Relative location data indicating a location of the first object in the first image data relative to at least one other object may be generated. The first embedding data, the second embedding data, and the relative location data may be input into a multi-modal transformer model. The multi-modal transformer model may determine that the first natural language data relates to the first object.

Natural language selection of objects in image data

Techniques for using a shared encoder and multiple different decoders for natural language understanding (NLU) tasks are described. The individual decoders are configured to perform different tasks using the output from one shared encoder. The decoders can process with respect to different domains and different languages. Using the shared encoder can reduce computation time during runtime. Using the shared encoder can reduce training costs (e.g., time and resources) when the system is updated to incorporate additional intents and entities. The system employs an attention mechanism to extract encoded representation data that can be used by the different decoders for its specific task.

Shared encoder for natural language understanding processing

Techniques for performing spoken language understanding (SLU) processing are described. An SLU component may include an audio encoder configured to perform an audio-to-text processing task and an audio-to-NLU processing task. The SLU component may also include a joint decoder configured to perform the audio-to-text processing task, the audio-to-NLU processing task and a text-to-NLU processing task. Input audio data, representing a spoken input, is processed by the audio encoder and the joint decoder to determine NLU data corresponding to the spoken input.

Speech processing

Techniques for tuning parameters for machine learning models are described. Different values for a parameter are tested to determine the value that results in an optimized model. A parameter value may be selected for testing using a search algorithm based on how the model performs with respect to other values for the parameter. Different values may be tested until a stopping criterion (such as time for testing, number of trials, amount of enhancement in performance, etc.) is met. In some embodiments, the techniques may be used to determine parameter values for natural language processing models.

Model configuration

Growing community of inventors

Yorktown Heights, NY, United States of America

Wael Hamza