Amazon Technologies, Inc.

David Paul Ramos

Matthew Joseph Cole

Avnish Sikka

Yue Liu

Matthew Daniel Hart

Sonjeev Jahagirdar

Emilie Noelle McConville

Utkarsh Prateek

Ankur Datta

Shiv Naga Prasad Vitaladevuni

Chang Yuan

Bhavesh Anil Doshi

Rakesh Madhavan Nambiar

Shuang Wu

Junxiong Jia

Thibaud Senechal

Matias Omar Gregorio Benitez

Prashant Hegde

Qingfeng Yu

Laura Varnum Finney

Ameya Ashok Limaye

Quan Wang

Daniel Makoto Willenson

Michael Vanne

Keith Harrison Goodman

Lukas Leon Brower

Vikas Chawla

Tonytip Ketudat

Techniques for performing multi-stage entity resolution (ER) processing are described. A system may determine a portion of a user input corresponding to an entity name, and may request an entity provider component to perform a search to determine one or more entities corresponding to the entity name. The preliminary search results may be sent to a skill selection component for processing, while the entity provider component performs a complete search to determine entities corresponding to the entity name. A selected skill component may request the complete search results to perform its processing, including determining an output responsive to the user input.

Multi-stage entity resolution

Techniques for performing entity resolution as part of natural language understanding processing are described. During offline operations, a system may convert text (representing entities known to the system) into audio of various languages. The languages into which the text is converted may depend on the location where the entity is likely to be spoken by users of the system. At runtime, the system processes a user input using text-based entity resolution. If text-based entity resolution fails, the system may identify user speech corresponding to an entity to be resolved, and attempt to phonetically match the user speech to the audio of the known entities. Results of the phonetic entity resolution may then used by downstream components, such as skills.

Language agnostic phonetic entity resolution

Disclosed are techniques for providing additional information for text in an image. In some implementations, a computing device receives an image including text. Optical character recognition (OCR) is performed on the image to produce recognized text. A word or a phrase is selected from the recognized text for providing additional information. One or more potential meanings of the selected word or phrase are determined. One of the potential meanings is selected based on other text in the image. A source of additional information corresponding to the selected meaning is selected for providing the additional information to a user's device.

Providing additional information for text in an image

Disclosed are techniques for merging optical character recognized (OCR'd) text from frames of image data. In some implementations, a device sends frames of image data to a server, where each frame includes at least a portion of a captured textual item. The server performs optical character recognition (OCR) on the image data of each frame. When OCR'd text from respective frames is returned to the device from the server, the device can perform matching operations on the text, for instance, using bounding boxes and/or edit distance processing. The device can merge any identified matches of OCR'd text from different frames. The device can then display the merged text with any corrections.

Merging optical character recognized text from frames of image data

A multi-orientation text detection method and associated system is disclosed that utilizes orientation-variant glyph features to determine a text line in an image regardless of an orientation of the text line. Glyph features are determined for each glyph in an image with respect to a neighboring glyph. The glyph features are provided to a learned classifier that outputs a glyph pair score for each neighboring glyph pair. Each glyph pair score indicates a likelihood that the corresponding pair of neighboring glyphs form part of a same text line. The glyph pair scores are used to identify candidate text lines, which are then ranked to select a final set of text lines in the image.

Text detection using features associated with neighboring glyph pairs

Disclosed are techniques for recognizing text from one or more frames of image data using contextual information. In some implementations, image data including a captured textual item is processed to identify an entity in the image data. A context can be selected using the entity, where the context corresponds to a dictionary. Text in the captured textual item can be identified using the dictionary. The identified text can be output to a display device.

Recognizing text from frames of image data using contextual information

Embodiments of the subject technology provide for a hybrid OCR approach which combines server and device side processing that can offset disadvantages of performing OCR solely on the server side or the device side. More specifically, the subject technology utilizes image characteristics such as glyph details and image quality measurements to opportunistically schedule OCR processing on the mobile device and/or server. In this regard, text extracted by a 'faster' OCR engine (e.g., one with less latency) is displayed to a user, which is then updated by the result of a more accurate OCR engine (e.g., an OCR engine provided by the server). This approach allows factoring in additional parameters such as network latency and user preference for making scheduling decisions. Thus, the subject technology may provide significant gains in terms of reduced latency and increased accuracy by implementing one or more techniques associated with this hybrid OCR approach.

Hybrid optical character recognition

Approaches to enable a computing device, such as a phone or tablet computer, to detect when text contained in an image captured by the camera is sufficiently close to the edge of the screen and to infer whether the text is likely to be cut off by the edge of the screen such that the text contained in the image is incomplete. If the incomplete text corresponds to actionable text associated with a function that can be invoked on the computing device, the computing device may wait until the remaining portion of the actionable text is captured by the camera and made available for processing before invoking the corresponding function on the computing device.

Text detection near display screen edge

Embodiments of the subject technology provide for determining a region of a first acquired image based at least on a viewing mode and a set of respective positions of graphical elements to decrease the pre-processing time and perceived latency for the first image. One or more regions of text in the first image are detected, and a set of regions of text that overlap with the region of the image is determined and pre-processed. The subject technology may then pre-process an entirety of a subsequent image (e.g., to pick up missing text from the region of the first image). Thus, additional OCR results may be provided to the user by using the subsequent image(s) and merging subsequent results with previous results from the first image.

Optimizing pre-processing times for faster response

The recognition of text in an acquired image is improved by using general and type-specific heuristics that can determine the likelihood that a portion of the text is truncated at an edge of an image, frame, or screen. Truncated text can be filtered such that the user is not provided with an option to perform an undesirable task, such as to dial an incorrect number or connect to an incorrect Web address, based on recognizing an incomplete text string. The general and type-specific heuristics can be combined to improve confidence, and the image data can be pre-processed on the device before processing with an optical character recognition (OCR) engine. Multiple frames can be analyzed to attempt to recognize words or characters that might have been truncated in one or more of the frames.

Text recognition near an edge

Disclosed are techniques for providing additional information for text in an image. In some implementations, a computing device receives an image including text. Optical character recognition (OCR) is performed on the image to produce recognized text. One or more topics corresponding to the recognized text is determined. A word or a phrase is selected from the recognized text for providing additional information. One or more potential meanings of the selected word or phrase are determined. One of the potential meanings is selected using the one or more topics. A source of additional information corresponding to the selected meaning is selected for providing the additional information to a user's device.

Various embodiments enable regions of text to be identified in an image captured by a camera of a computing device for preprocessing before being analyzed by a visual recognition engine. For example, each of the identified regions can be analyzed or tested to determine whether a respective region contains a quality associated with poor text recognition results, such as poor contrast, blur, noise, and the like, which can be measured by one or more algorithms. Upon identifying a region with such a quality, an image quality enhancement can be automatically applied to the respective region without user instruction or intervention. Accordingly, once each region has been cleared of the quality associated with poor recognition, the regions of text can be processed with a visual recognition algorithm or engine.

Growing community of inventors

Cambridge, MA, United States of America

David Paul Ramos