Amazon Technologies, Inc.

Jaechul Kim

Daniel Bibireata

Nishitkumar Ashokkumar Desai

Dilip Kumar

Gerard Guy Medioni

Jayakrishnan Kumar Eledath

Kushagra Srivastava

Yasser Baseer Asmi

Xiaofeng Ren

Emilio Ian Maldonado

Hoi Cheung Pang

Kartik Muktinutalapati

Shaonan Zhang

Frank Florian Liberato, Jr

Sudarshan Narasimha Raghavan

Ammar Chinoy

Muralidhar Koka

Hao Jiang

Chen Xu

Where an event is determined to have occurred at a location within a vicinity of a plurality of actors, imaging data captured using cameras having the location is processed using one or more machine learning systems or techniques operating on the cameras to determine which of the actors is most likely associated with the event. For each relevant pixel of each image captured by a camera, the camera returns a set of vectors extending to pixels of body parts of actors who are most likely to have been involved with an event occurring at the relevant pixel, along with a measure of confidence in the respective vectors. A server receives the vectors from the cameras, determines which of the images depicted the event in a favorable view, based at least in part on the quality of such images, and selects one of the actors as associated with the event accordingly.

Associating events with actors using digital imagery and machine learning

Where an event is determined to have occurred at a location within a vicinity of a plurality of actors, imaging data captured using cameras having the location is processed using one or more machine learning systems or techniques operating on the cameras to determine which of the actors is most likely associated with the event. For each relevant pixel of each image captured by a camera, the camera returns a set of vectors extending to pixels of body parts of actors who are most likely to have been involved with an event occurring at the relevant pixel, along with a measure of confidence in the respective vectors. A server receives the sets of vectors from the cameras, determines which of the images depicted the event in a favorable view, based at least in part on the quality of such images, and selects one of the actors as associated with the event accordingly.

Sensors in a facility generate sensor data associated with a region of the facility, which can be used to determine a 3D location of an object in the facility. Some sensors may sense overlapping regions of the facility. For example, a first sensor may generate data associated with a first region of the facility, while a second sensor may generate data associated with a second region of the facility that partially overlaps the first region. Sensors may fail at times as determined from sensor output data or status data. In response to identifying a failed sensor, an undetected region corresponding to the failed sensor is identified, as well as a substitute sensor that partially senses the undetected region. Sensor data from the substitute sensor, such as 2D data, is acquired and used to estimate a 3D location of an object in the undetected region.

Management of sensor failure in a facility

During operation, a facility may utilize many sensors, such as cameras, to generate sensor data. The sensor data may be processed by an inventory management system to track objects, determine the occurrence of events at the facility, and so forth. At any given time, some of these sensors may fail to provide timely data, may fail to provide any data, may generate inaccurate data, and so forth. Described are techniques to determine failure of sensors and adjust operation of the inventory management system to maintain operability during sensor failure.

Described is a multiple-camera system and process for detecting a user within a materials handling facility and tracking a position of the user as the user moves through the materials handling facility. In one implementation, a plurality of depth sensing cameras are positioned above a surface of the materials handling facility and oriented to obtain an overhead view of the surface of the materials handling facility, along with any objects (e.g., users) on the surface of the materials handling facility. The depth information from the cameras may be utilized to detect objects on the surface of the materials handling facility, track a movement of those objects and determine if those objects are users.

Using depth sensing cameras positioned overhead to detect and track a movement of a user within a materials handling facility

Described is a multiple-camera system for use in capturing images of users within a materials handling facility and processing those images to monitor the movement of users. For large materials handling facilities, a large number of cameras may be required to monitor the facility. Processing of the data generated from a large number of cameras becomes difficult. The implementations described herein include a hierarchy that allows image data from any number of cameras within a materials handling facility to be processed without substantially increasing the processing time needed or sacrificing processing capabilities.

Camera hierarchy for monitoring large facilities

Growing community of inventors

Seattle, WA, United States of America

Jaechul Kim