Amazon Technologies, Inc.

Daniel Bibireata

Nishitkumar Ashokkumar Desai

Yasser Baseer Asmi

Emilio Ian Maldonado

Geoffrey Heller

Sudarshan Narasimha Raghavan

Ammar Chinoy

Frank Florian Liberato, Jr

Jaechul Kim

Jayakrishnan Kumar Eledath

Richard Howard Suplee, Iii

Dilip Kumar

Muralidhar Koka

Hao Jiang

Gerard Guy Medioni

Chang Yuan

Kushagra Srivastava

Louis LeRoi LeGrand, Iii

Francislav Petrov Penov

Xiaofeng Ren

Kartik Muktinutalapati

Hoi Cheung Pang

Shaonan Zhang

Leo Benedict Baldwin

Aaron Michael Donsbach

Kenneth Mark Karakotsios

Jason George McHugh

Volodymyr V Ivanchenko

Gopi Prashanth Gopal

Timothy Youngjin Sohn

Liefeng Bo

Paul Eugene Munger

Beryl Tomay

Kah Kuen Fu

I-Hung Wang

Hilliard Bruce Siegel

Wei Li

Oleg Rybakov

Prafulla Jinendra Masalkar

Avinash Aghoram Ravichandran

David Allen Smith

Pahal Kamlesh Dalal

Joachim Sebastian Stahl

Christopher S A MacGregor

Ajay Kumar Mishra

Honghong Peng

Brian R Williams

Wei Zhang

Charles P Vaughn

Neil David Cooper

Varun Narang

Laura Varnum Finney

Chen Xu

Todd Nelson Schoepflin

Min Xu

Kevin Kar Wai Lai

Bradley David Volen

Saurabh Verma

Matthew J Rubens

Tidiness data associated with an inventory location, such as a shelf, indicates whether an item returned to the inventory location is designated for storage at the inventory location. When an item is returned to the inventory location and a user is determined to be near the inventory location, the weight change detected at the inventory location is compared to a corresponding weight of each item in the custody of the user, such as items within cart state data associated with the user. Differences in weight between the weight change and the corresponding weight of each item in the custody of the user are calculated. Such differences may then be compared to one or more metrics, such as threshold values, to determine how well the top candidate item in the custody of the user matches the type of item designated for storage at the inventory location.

Generating tidiness data associated with items at an inventory location

Described is a multiple-camera system and process for determining an item involved in an event. For example, when a user picks an item or places an item at an inventory location, image information for the item may be obtained and processed to identify the item involved in the event and associate that item with the user.

Store tracking system

Where an event is determined to have occurred at a location within a vicinity of a plurality of actors, imaging data captured using cameras having the location is processed using one or more machine learning systems or techniques operating on the cameras to determine which of the actors is most likely associated with the event. For each relevant pixel of each image captured by a camera, the camera returns a set of vectors extending to pixels of body parts of actors who are most likely to have been involved with an event occurring at the relevant pixel, along with a measure of confidence in the respective vectors. A server receives the vectors from the cameras, determines which of the images depicted the event in a favorable view, based at least in part on the quality of such images, and selects one of the actors as associated with the event accordingly.

Associating events with actors using digital imagery and machine learning

Described is a system and method for collecting item and user information and utilizing that information to determine materials handling facility patterns, item trends and inventory location trends. User monitoring data for users located in a materials handling facility may include information identifying inventory locations approached by users, gaze directions of users, user dwell times, an identification of items picked by users, an identification of items placed by users, and/or in-transit times for items. This information may be aggregated and processed to determine materials handling facility patterns, item trends and/or inventory location trends.

Determining trends from materials handling facility information

Determining an item involved in an event at an event location

Where an event is determined to have occurred at a location within a vicinity of a plurality of actors, imaging data captured using cameras having the location is processed using one or more machine learning systems or techniques operating on the cameras to determine which of the actors is most likely associated with the event. For each relevant pixel of each image captured by a camera, the camera returns a set of vectors extending to pixels of body parts of actors who are most likely to have been involved with an event occurring at the relevant pixel, along with a measure of confidence in the respective vectors. A server receives the sets of vectors from the cameras, determines which of the images depicted the event in a favorable view, based at least in part on the quality of such images, and selects one of the actors as associated with the event accordingly.

Sensors in a facility generate sensor data associated with a region of the facility, which can be used to determine a 3D location of an object in the facility. Some sensors may sense overlapping regions of the facility. For example, a first sensor may generate data associated with a first region of the facility, while a second sensor may generate data associated with a second region of the facility that partially overlaps the first region. Sensors may fail at times as determined from sensor output data or status data. In response to identifying a failed sensor, an undetected region corresponding to the failed sensor is identified, as well as a substitute sensor that partially senses the undetected region. Sensor data from the substitute sensor, such as 2D data, is acquired and used to estimate a 3D location of an object in the undetected region.

Management of sensor failure in a facility

Described are systems and techniques to generate data for display to present visualizations of data acquired from sensors in a facility. The data visualizations may be used to develop, configure, administer, or otherwise support operation of the facility. In one implementation, the visualization may include a view incorporating aggregated images acquired from multiple cameras, depth data, tracking information about objects in the facility, and so forth. An analyst may use the data visualization to determine occurrence of an action in the facility such as a pick of an item, place of an item, what item was involved with an action, what user was involved with the action, and so forth. Based on the information presented by the data visualization, changes may be made to data processing parameters.

Visualization system for sensor data and facility data

A user may pick an item from a first inventory location, such as in a lane on a shelf, and may return it another location that is assigned to another type of item. Described are techniques to generate tidiness data that is indicative of whether an item has been returned to an inventory location assigned to that type of item. As items are taken, information about the type of item taken and its weight are stored. When an increase in weight at a lane indicates a return of an item to the lane, the weight of the return is compared to the stored weight of the items previously taken by a user. If the weights correspond to within a threshold value, the type of item associated with the stored weight is deemed to be returned and tidiness data indicative of a tidy return of the item to its appointed lane may be generated.

Determination of untidy item return to an inventory location using weight

Described is a multiple-camera system for use in capturing images of users within a materials handling facility and processing those images to monitor the movement of users. For large materials handling facilities, a large number of cameras may be required to monitor the facility. Processing of the data generated from a large number of cameras becomes difficult. The implementations described herein include a hierarchy that allows image data from any number of cameras within a materials handling facility to be processed without substantially increasing the processing time needed or sacrificing processing capabilities.

Monitoring locations within a materials handling facility

During operation, a facility may utilize many sensors, such as cameras, to generate sensor data. The sensor data may be processed by an inventory management system to track objects, determine the occurrence of events at the facility, and so forth. At any given time, some of these sensors may fail to provide timely data, may fail to provide any data, may generate inaccurate data, and so forth. Described are techniques to determine failure of sensors and adjust operation of the inventory management system to maintain operability during sensor failure.

Described is a multiple-camera system and process for detecting a user within a materials handling facility and tracking a position of the user as the user moves through the materials handling facility. In one implementation, a plurality of depth sensing cameras are positioned above a surface of the materials handling facility and oriented to obtain an overhead view of the surface of the materials handling facility, along with any objects (e.g., users) on the surface of the materials handling facility. The depth information from the cameras may be utilized to detect objects on the surface of the materials handling facility, track a movement of those objects and determine if those objects are users.

Using depth sensing cameras positioned overhead to detect and track a movement of a user within a materials handling facility

Growing community of inventors

Seattle, WA, United States of America

Daniel Bibireata