Amazon Technologies, Inc.

Raytheon Bbn Technologies Corp.

Pradeep Natarajan

Vivek Yadav

Rohit Prasad

Yue Wu

Kiana Hajebi

Premkumar Natarajan

Ayush Jaiswal

Yue Liu

Avnish Sikka

Qingfeng Yu

Shiv N Vitaladevuni

Aayush Gupta

Xing Liu

Shih-Fu Chang

Shuang Wu

Xu Zhang

Xiaodan Zhuang

Manasvi Tickoo

Varsha Hedau

Jianwei Feng

Devansh Arpit

Walter Andrews

Devices and techniques are generally described for inference reduction in natural language processing using semantic similarity-based caching. In various examples, first automatic speech recognition (ASR) data representing a first natural language input may be determined. A cache may be searched using the first ASR data. A first skill associated with the first ASR data may be determined from the cache. In some examples, first intent data representing a semantic interpretation of the first natural language input data may be determined by using a first natural language process associated with the first skill.

Natural language processing

Techniques for masking images based on a particular task are described. A system masks portions of an image that are not relevant to a particular task, thus, reducing the amount of data used by applications for image processing tasks. For example, images to be processed using a hair color classification model are masked so that only portions that show the person's hair are available for the model to analyze. The system configures different masker components to mask images for different tasks. A masker component can be implemented at a user device to mask images prior to sending to an application/task-specific model.

Task-based image masking

Devices and techniques are generally described for class-agnostic object detection. In some examples, a first frame of image data comprising a first plurality of pixels may be received. First class-agnostic feature data representing the first plurality of pixels may be generated. A first object detection component may be used to determine that the first plurality of pixels corresponds to an arbitrary object represented in the first frame of image data based at least in part on the first class-agnostic feature data. Class-agnostic data indicating that the first plurality of pixels in the first frame of image data corresponds to the arbitrary object may be generated.

Class-agnostic object detection

An image processing system may receive image data from a camera of a user device and perform gaze prediction processing of the image data to predict one or more gaze patterns. The gaze prediction processing may include processing the image data using a neural network to detect faces and/or objects and generate an image feature map. The gaze prediction processing may include performing gaze direction prediction operations using the feature map and detected faces and/or objects to determine gaze direction probability data. The gaze prediction processing may include predicting a gaze pattern based on the gaze direction probability data and the image feature map. The gaze pattern may be short-term (e.g., atomic-level) or long-term (e.g., event-level).

Gaze prediction

Devices and techniques are generally described for three dimensional mesh generation. In various examples, first two-dimensional (2D) image data representing a human may be received. In various further examples, bounding box data identifying a location of the human in the first 2D image data and joint data identifying locations of joints of the human may be received. Second 2D image data representing a cropped portion of the human may be generated using the bounding box data and the joint data. A three-dimensional (3D) mesh prediction model may be used to determine a pose, a shape, and a projection matrix for the human. The 3D mesh prediction model may be used to determine a transformed projection matrix for the portion of the human represented in the second 2D image data.

Three-dimensional mesh generation

A system to recognize text or symbols contained in a captured image using machine learning models leverages context information about the image to improve accuracy. Contextual information is determined for the entire image, or spatial regions of the images, and is provided to a machine learning model when a determination is made as to whether a region does or does not contain text or symbols. Associating features related to the larger context with features extracted from regions potentially containing text or symbolic content provides an incremental improvement of results obtained using machine learning techniques.

Leveraging image context for improved glyph classification

A system to select video frames for optical character recognition (OCR) based on feature metrics associated with blur and sharpness. A device captures a video frame including text characters. An edge detection filter is applied to the frame to determine gradient features in perpendicular directions. An 'edge map' is created from the gradient features, and points along edges in the edge map are identified. Edge transition widths are determined at each of the edge points based in local intensity minimum and maximum on opposite sides of the respective edge point in the frame. Sharper edges have smaller edge transition widths than blurry images. Statistics are determined from the edge transition widths, and the statistics are processed by a trained classifier to determine if the frame is or is not sufficiently sharp for text processing.

Sharpness-based frame selection for OCR

A method for feature transformation of a data set includes: receiving a data set including original feature samples with corresponding class labels; splitting the data set into a direction optimization set and a training set; using the direction optimization set to calculate an optimum transformation vector that maximizes inter-class separability and minimizes intra-class variance of the feature samples with respect to corresponding class labels; using the optimum transformation vector to transform the rest of the original feature samples of the data set to new feature samples with enhanced discriminative characteristics; and training a classifier using the new feature samples, wherein the method is performed by one or more processors.

Class discriminative feature transformation

A system to recognize text, objects, or symbols in a captured image using machine learning models reduces computational overhead by generating a plurality of thumbnail versions of the image at different downscaled resolutions and aspect ratios, and then processing the downscaled images instead of the entire image, or sections of the entire image. The downscaled images are processed to produce a combine feature vector characterizing the overall image. The combined feature vector is processed using the machine learning model.

Image processing using multiple aspect ratios

A process for training and optimizing a system to select video frames for optical character recognition (OCR) based on feature metrics associated with blur and sharpness. A set of image frames are subjectively labelled based on a comparison of each frame before and after binarization to determine to what degree text is recognizable in the binary image. A plurality of different sharpness feature metrics are generated based on the original frame. A classifier is then trained using the feature metrics and the subjective labels. The feature metrics are then tested for accuracy and/or correlation with subjective labelling data. The set of feature metrics may be refined based on which metrics produce the best results.

An approach to computation of kernel descriptors is accelerated using precomputed tables. In one aspect, a fast algorithm for kernel descriptor computation that takes O(1) operations per pixel in each patch, based on pre-computed kernel values. This speeds up the kernel descriptor features under consideration, to levels that are comparable with D-SIFT and color SIFT, and two orders of magnitude faster than STIP and HoG3D. In some examples, kernel descriptors are applied to extract gradient, flow and texture based features for video analysis. In tests of the approach on a large database of internet videos used in the TRECVID MED 2011 evaluations, the flow based kernel descriptors are up to two orders of magnitude faster than STIP and HoG3D, and also produce significant performance improvements. Further, using features from multiple color planes produces small but consistent gains.

Growing community of inventors

Lexington, MA, United States of America

Pradeep Natarajan