Amazon Technologies, Inc.

Nishitkumar Ashokkumar Desai

Jayakrishnan Kumar Eledath

Daniel Bibireata

Gerard Guy Medioni

Dilip Kumar

Sudarshan Narasimha Raghavan

Emilio Ian Maldonado

Yasser Baseer Asmi

Kushagra Srivastava

Nikhil Chacko

Pahal Kamlesh Dalal

Ammar Chinoy

Jaechul Kim

Kartik Muktinutalapati

Gopi Prashanth Gopal

Waqas Syed Ahmed

Danny Guan

Liefeng Bo

Connor Spencer Blue Worley

Frank Florian Liberato, Jr

Behjat Siddiquie

Petko Tsonev

Muralidhar Koka

Hoi Cheung Pang

Claire Law

Hao Jiang

Jason Michael Famularo

Paul Eugene Munger

Maren Marie Costa

Thomas Meilandt Mathiesen

Jean Laurent Guigues

Alessandro Bergamo

Casey Louis Thurston

Robert Crandall

Kaustav Kundu

Christopher Richard Fescoe

Marian Nasr Amin George

Jue Wang

Hue Tuan Thi

Shaonan Zhang

Bharat Singh

Xiaofeng Ren

Robert Matthias Steele

Jiajun Zhao

Joseph M Alyea

Chuhang Zou

Longlong Zhu

Lu Xia

Rakesh Ramakrishnan

Matthew Fletcher Harrison

Jingjing Liu

Paul Ignatius Dizon Echevarria

Keshav Nand Tripathi

Ivan Stankovic

Geoffrey A Franz

I-Hung Wang

David Allen Smith

Keunhong Park

Kevin Kar Wai Lai

Min Xu

This disclosure describes systems and techniques for identifying events that occur within an environment using image data captured at the environment. For example, one or more cameras may generate image data representative of a user interacting with an item on the shelf. This image data may be used to generate feature data associated with the user and the item, which may be analyzed by one or more classifiers for identifying an interaction between the user and the item. The systems and techniques may then generate interaction data, which in turn may be analyzed by one or more additional classifiers for identifying an event, such as the user picking a particular item from the shelf within the environment. Event data indicative of the event may then be used to update a virtual cart of the user.

System and method for vision-based event detection

This disclosure describes techniques for updating planogram data associated with a facility. The planogram may indicate inventory locations within the facility for various types of items supported by product fixtures. In particular an image of a product fixture is analyzed to identify image segments corresponding to product groups, where each product group consists of instances of the same product and each image segment corresponds to a group of image points. Image data is further analyzed to determine coordinates of the points of each image segment. A product space corresponding to the product group is then defined based on the coordinates of the points of the product group. In some cases, for example, a product space may be defined in terms of the coordinates of the corners of a rectangular bounding box or volume.

Image-based detection of planogram product spaces

This disclosure describes techniques for updating planogram data associated with a facility. The planogram may indicate, for different shelves and other inventory locations within the facility, which items are on which shelves. For example, the planogram data may indicate that a particular item is located on a particular shelf. Therefore, when a system identifies that a user has taken an item from that shelf, the system may update a virtual cart of that user to indicate addition of the particular item. In some instances, however, a new item may be stocked on the example shelf instead of a previous item. The techniques described herein may use sensor data generated in the facility to identify this change and update the planogram data to indicate an association between the shelf and the new item.

Inferring facility planograms

This disclosure is directed to techniques in which a first user in an environment scans visual indicia associated with an item, such as a barcode, before handing the item to a second user. One or more computing devices may receive an indication of the scan, retrieve image data of the interaction from a camera within the environment, identify the user that received the item, and update a virtual cart associated with the second user to indicate addition of the item.

Analyzing sensor data to identify events

Described is a multiple-camera system and process for determining an item involved in an event. For example, when a user picks an item or places an item at an inventory location, image information for the item may be obtained and processed to identify the item involved in the event and associate that item with the user.

Store tracking system

Where an event is determined to have occurred at a location within a vicinity of a plurality of actors, imaging data captured using cameras having the location is processed using one or more machine learning systems or techniques operating on the cameras to determine which of the actors is most likely associated with the event. For each relevant pixel of each image captured by a camera, the camera returns a set of vectors extending to pixels of body parts of actors who are most likely to have been involved with an event occurring at the relevant pixel, along with a measure of confidence in the respective vectors. A server receives the vectors from the cameras, determines which of the images depicted the event in a favorable view, based at least in part on the quality of such images, and selects one of the actors as associated with the event accordingly.

Associating events with actors using digital imagery and machine learning

Commercial interactions with non-discretized items such as liquids in carafes or other dispensers are detected and associated with actors using images captured by one or more digital cameras including the carafes or dispensers within their fields of view. The images are processed to detect body parts of actors and other aspects therein, and to not only determine that a commercial interaction has occurred but also identify an actor that performed the commercial interaction. Based on information or data determined from such images, movements of body parts associated with raising, lowering or rotating one or more carafes or other dispensers may be detected, and a commercial interaction involving such carafes or dispensers may be detected and associated with a specific actor accordingly.

Detecting interactions with non-discretized items and associating interactions with actors using digital images

Described is a system and method for collecting item and user information and utilizing that information to determine materials handling facility patterns, item trends and inventory location trends. User monitoring data for users located in a materials handling facility may include information identifying inventory locations approached by users, gaze directions of users, user dwell times, an identification of items picked by users, an identification of items placed by users, and/or in-transit times for items. This information may be aggregated and processed to determine materials handling facility patterns, item trends and/or inventory location trends.

Determining trends from materials handling facility information

A system may generate image data of users within a facility. Such users may be part of a group that is unknown to the system. The system can predict group data for two or more users based on a resemblance or the users being within a threshold distance of each other. However, if a confidence level associated with the predicted group data is below a threshold value, the group data is deemed unreliable and assistance from an associate is deemed necessary. A user interface that includes a portion of the image data, information about the predicted group data, and other interface elements is presented to the associate via a display. Based on the input data received from the associate, the group data can be confirmed or rejected. If the group data is confirmed, an association is made between the users and a group identifier.

Presentation of a user interface for confirming unreliable group data

Determining an item involved in an event at an event location

Growing community of inventors

Redmond, WA, United States of America

Nishitkumar Ashokkumar Desai