Amazon Technologies, Inc.

Other

Ejaz Ahmed

Oded Maron

Jeremy Samuel De Bonet

Mirko Ristivojevic

Daniel Ron

Kilho Son

Austen Groener

Gregory Donald Hager

Jonathan Evan Cohn

Abhay Doke

Sajjad Beygiharchegani

Gopi Prashanth Gopal

Jiajun Zhao

Andre David Brown

Hue Tuan Thi

Vinod Krishnan Kulathumani

Chris Broaddus

Longlong Zhu

Venkataraman Santhanam

Xiaoxia Shi

Michael Derryberry

Yuzhou Feng

Shamil Abzalov

Mohsen Malmir

This disclosure is directed to an item-identifying, mobile cart that may be utilized by a user in a materials handling facility to automatically identify items that the user places into a basket of the cart. The cart may include cameras for detecting activity near a basket of the cart, such as a user placing an item into the basket. The cart or a computing device communicatively coupled to the cart may use a trained machine-learning model to identify the activity, and may improve the accuracy of the model over time using real-world data of users operating carts in respective facilities.

Conserving power in item-identifying carts

This disclosure describes, in part, techniques for identifying interactions and events associated with inventory locations. For instance, system(s) may receive image data representing a user interacting with an inventory location. The system(s) may then generate heatmap data indicating a first portion of the image data that represents the inventory location and feature data indicating a second portion of the image data that represents the user. Next, the system(s) may analyze the heatmap data with respect to the feature data to determine that the second portion of the image data corresponds to the first portion of the image data. As such, the system(s) may determine that the user is interacting with the inventory location. Based on the determination, the system(s) may analyze the first portion of the image data to identify an event that occurs at the inventory location, such as the user removing an item.

Detecting interactions with inventory locations

This disclosure is directed to, in part, a processing pipeline for detecting predefined activity using image data, identifying an item-of-interest and a location of the item-of-interest across frames of the image data, determining a trajectory of the item-of-interest, and determining an identifier of the item-of-interest and an action taken with respect to the item-of-interest. The processing pipeline may utilize one or more trained classifiers and, in some instances, additional data to identify items that are placed into or removed from a tote (e.g., basket, cart, or other receptacle) by users in material handling facilities as the users move around the material handling facilities.

Using sensor data to determine activity

This disclosure describes, in part, systems for enabling physical retail stores and other facilities to implement both automated- and manual-checkout techniques for customers of the stores and/or facilities. For example, the described systems may enable a retail store to implement technology where users are able to pick items from shelves and other inventory locations and exit the store without performing manual checkout of the items, as well as technology to allow users to pay for their items using point-of-sale (POS) and/or other manual-checkout techniques. The systems described herein also generate associations between users and carts within the stores to prevent fraudulent transactions within these hybrid environments.

Associating carts and users in hybrid environments

This disclosure describes, in part, techniques for collecting image data representing item identifiers, such as barcodes. For instance, system(s) may receive image data representing images, where the images depict at least a portion of an identifier located on an item. The system(s) may then identify a first portion of the image data representing an image that that is associated a low confidence level. Next, the system(s) may identify a second portion of the image data representing additional images that are associated high confidence levels. Using results for this the second portion of the image data, the system(s) may determine a ground truth result for the first portion of the image data. The system(s) may then store, in one or more databases, data representing the ground truth result in association with the first portion of the image data.

Techniques for storing images depicting identifiers

This disclosure describes, in part, a mobile apparatus for identifying items within a facility using an omnidirectional imaging system. For instance, the mobile apparatus may include a main frame, a chassis attached to the main frame, a basket that attaches to the chassis in order to prevent the basket from contacting the main frame, and an omnidirectional imaging system providing image data in and around the mobile apparatus. A user may place item(s) within a receptacle of the basket. The mobile apparatus may further include a handlebar module attached to the main frame, the handlebar module including the omnidirectional imaging system and a computing system for identifying items and events in and around the cart.

Omnidirectional item identifying mobile apparatus

This disclosure is directed to using cascading algorithms to automatically identify items placed in a tote or other receptacle utilized by users in material handling facilities as the users move around the facilities. A tote may store a database or 'gallery' of item representations for all of the items that are stored in the facility that a user may place in their totes. The tote may use multiple algorithms in a cascading manner to analyze the gallery of item representations in order to iteratively narrow the search space of item representations in the gallery to determine which of the items was placed in the tote by a user. Upon identifying the item placed in the tote, the tote may add an item identifier for the item to a virtual listing of item identifiers representing items previously placed in the tote.

Identifying items using cascading algorithms

This disclosure is directed to item-identifying carts that may generate and store image data of items being placed into, or removed from, the carts in the event that a cart is unable to recognize an item. For example, the cart may analyze frames of generated image data, representing item(s) being placed into or removed from the cart, to determine which one or more frames to store. If the cart is later unable to identify such an item, the cart may retrieve one or more of the stored frames, generate a user interface (UI) that includes these frame(s) and a request that a user operating the cart to help identify the item, and present the UI on a display of the cart.

Resolving events in item-identifying carts

A tag with sufficient variability is generated. The tag facilitates quick location and processing within a facility. During generation, a matrix representative of a possible tag may be tested for sufficient variability. A two-dimensional matrix representing the possible tag may include different bit values at different positions. A variability metric of the matrix is performed to check the variability between adjacent elements of the matrix. This variability metric of the matrix is then compared against a variability threshold value. If the variability metric of the matrix falls below the threshold value, the matrix is flagged as unusable. If the variability metric of the matrix exceeds the threshold value, the matrix can be used to generate a tag with sufficient variability. Once generated, the tag can be presented and affixed to an item or object to help identify and locate the item within the facility.

System for generating tags

This disclosure describes using trained classifiers and additional data to identify items placed into a tote. The techniques may use a first classifier on image data representing an item to determine a region associated with a barcode of the item and may use one or more second classifiers to analyze the region for determining a candidate barcode. The techniques may then determine whether the candidate barcode corresponds to the actual barcode of the item. If so, the techniques may update a virtual cart associated with a user account to indicate the acquisition of the item. If not, then the techniques may use the additional data for identifying the item. For example, the techniques determine a current location of a tote for determining items that are within a threshold distance of the location and may use this information, along with the candidate barcode, to identify the item placed into the tote.

Identifying item barcodes

A tag comprises cells represented by marks, such as white and black, arranged in a predetermined pattern. Reference cells are used to determine the edges of the tag and orientation, while others are used to encode data and error correction data. During processing, a tag is detected based on a variability metric within a given region of an image exceeding a threshold value. Once detected, the orientation of the tag is determined using the reference cells. Confidence values, indicative of a likelihood that the cell is a particular cell value, such as white or black, are determined for each cell in a first set of cell values. If the first set is invalid, the cell values for a set of lowest confidence cell values are inverted to produce a second set of cell values. The second set is tested, and if valid, is used to produce output data.

Growing community of inventors

Marlborough, MA, United States of America

Ejaz Ahmed