Amazon Technologies, Inc.

Jayakrishnan Kumar Eledath

Nishitkumar Ashokkumar Desai

Gerard Guy Medioni

Dilip Kumar

Kushagra Srivastava

Gopi Prashanth Gopal

Liefeng Bo

Nikhil Chacko

Behjat Siddiquie

Pahal Kamlesh Dalal

Hoi Cheung Pang

Petko Tsonev

Daniel Bibireata

Paul Eugene Munger

Jue Wang

Connor Spencer Blue Worley

Jean Laurent Guigues

Tian Lan

Kartik Muktinutalapati

Claire Law

Jaechul Kim

Robert Crandall

Alessandro Bergamo

Hue Tuan Thi

Kaustav Kundu

Chuhang Zou

Shaonan Zhang

Marian Nasr Amin George

Bharat Singh

Robert Matthias Steele

Shane Michael Wilson

Samuel Nathan Hallman

Lu Xia

Matthew Norman

Rakesh Ramakrishnan

Michael Francis O'Malley

Geoffrey A Franz

Paul Ignatius Dizon Echevarria

Matthew Fletcher Harrison

Leonid Pishchulin

Chris Broaddus

Keunhong Park

Hui Liang

Dinesh Reddy Narapureddy

Jingjing Liu

Keshav Nand Tripathi

This disclosure describes techniques for updating planogram data associated with a facility. The planogram may indicate inventory locations within the facility for various types of items supported by product fixtures. In particular an image of a product fixture is analyzed to identify image segments corresponding to product groups, where each product group consists of instances of the same product and each image segment corresponds to a group of image points. Image data is further analyzed to determine coordinates of the points of each image segment. A product space corresponding to the product group is then defined based on the coordinates of the points of the product group. In some cases, for example, a product space may be defined in terms of the coordinates of the corners of a rectangular bounding box or volume.

Image-based detection of planogram product spaces

This disclosure describes techniques for updating planogram data associated with a facility. The planogram may indicate, for different shelves and other inventory locations within the facility, which items are on which shelves. For example, the planogram data may indicate that a particular item is located on a particular shelf. Therefore, when a system identifies that a user has taken an item from that shelf, the system may update a virtual cart of that user to indicate addition of the particular item. In some instances, however, a new item may be stocked on the example shelf instead of a previous item. The techniques described herein may use sensor data generated in the facility to identify this change and update the planogram data to indicate an association between the shelf and the new item.

Inferring facility planograms

This disclosure describes, in part, systems for enabling physical retail stores and other facilities to implement both automated- and manual-checkout techniques for customers of the stores and/or facilities. For example, the described systems may enable a retail store to implement technology where users are able to pick items from shelves and other inventory locations and exit the store without performing manual checkout of the items, as well as technology to allow users to pay for their items using point-of-sale (POS) and/or other manual-checkout techniques. The systems described herein thus enable hybrid retail facilities, as opposed to a retail facility that is either entirely traditional or entirely enabled for automated checkout.

Hybrid retail environments

Cameras having storage fixtures within their fields of view are programmed to capture images and process clips of the images to generate sets of features representing product spaces and actors depicted within such images, and to classify the clips as depicting or not depicting a shopping event. Where consecutive clips are determined to depict a shopping event, features of such clips are combined into a sequence and transferred, along with classifications of the clips and a start time and end time of the shopping event, to a multi-camera system. A shopping hypothesis is generated based on such sequences of features received from cameras, along with information regarding items detected within the hands of such actors, to determine a summary of shopping activity by an actor, and to update a record of items associated with the actor accordingly.

Detecting interactions from features determined from sequences of images captured using one or more cameras

Tidiness data associated with an inventory location, such as a shelf, indicates whether an item returned to the inventory location is designated for storage at the inventory location. When an item is returned to the inventory location and a user is determined to be near the inventory location, the weight change detected at the inventory location is compared to a corresponding weight of each item in the custody of the user, such as items within cart state data associated with the user. Differences in weight between the weight change and the corresponding weight of each item in the custody of the user are calculated. Such differences may then be compared to one or more metrics, such as threshold values, to determine how well the top candidate item in the custody of the user matches the type of item designated for storage at the inventory location.

Generating tidiness data associated with items at an inventory location

Where an event is determined to have occurred at a location within a vicinity of a plurality of actors, imaging data captured using cameras having the location is processed using one or more machine learning systems or techniques operating on the cameras to determine which of the actors is most likely associated with the event. For each relevant pixel of each image captured by a camera, the camera returns a set of vectors extending to pixels of body parts of actors who are most likely to have been involved with an event occurring at the relevant pixel, along with a measure of confidence in the respective vectors. A server receives the vectors from the cameras, determines which of the images depicted the event in a favorable view, based at least in part on the quality of such images, and selects one of the actors as associated with the event accordingly.

Associating events with actors using digital imagery and machine learning

Commercial interactions with non-discretized items such as liquids in carafes or other dispensers are detected and associated with actors using images captured by one or more digital cameras including the carafes or dispensers within their fields of view. The images are processed to detect body parts of actors and other aspects therein, and to not only determine that a commercial interaction has occurred but also identify an actor that performed the commercial interaction. Based on information or data determined from such images, movements of body parts associated with raising, lowering or rotating one or more carafes or other dispensers may be detected, and a commercial interaction involving such carafes or dispensers may be detected and associated with a specific actor accordingly.

Detecting interactions with non-discretized items and associating interactions with actors using digital images

As agents move about a materials handling facility, tracklets representative of the position of each agent are maintained along with a confidence score indicating a confidence that the position of the agent is known. If the confidence score falls below a threshold level, image data of the agent associated with the low confidence score is obtained and processed to generate one or more embedding vectors representative of the agent at a current position. Those embedding vectors are then compared with embedding vectors of other candidate agents to determine a set of embedding vectors having a highest similarity. The candidate agent represented by the set of embedding vectors having the highest similarity score is determined to be the agent and the position of that candidate agent is updated to the current position, thereby re-identifying the agent.

Re-identification of agents using image analysis and machine learning

This disclosure describes systems and techniques for identifying events that occur within an environment using image data captured at the environment. For example, one or more cameras may generate image data representative of a user interacting with an item on the shelf. This image data may be used to generate feature data associated with the user and the item, which may be analyzed by one or more classifiers for identifying an interaction between the user and the item. The systems and techniques may then generate interaction data, which in turn may be analyzed by one or more additional classifiers for identifying an event, such as the user picking a particular item from the shelf within the environment. Event data indicative of the event may then be used to update a virtual cart of the user.

System and method for vision-based event detection

An imaging device may capture images of a scene, where the scene includes retroreflective materials. Where visual images and depth images are captured from a scene, and the depth images have ratios of supersaturated pixels that are less than a predetermined threshold, a location map of the scene is generated or updated based on the depth images. Where the ratios are greater than the predetermined threshold, the location map of the scene is generated or updated based on the visual images. Additionally, where each of a plurality of imaging devices detect concentrations of supersaturated pixels beyond a predetermined threshold or limit within their respective fields of view, an actor present on the scene may be determined to be wearing retroreflective material, or otherwise designated as a source of the supersaturation, and tracked with the scene based on coverage areas that are determined to have excessive ratios of supersaturated pixels.

Detecting and locating actors in scenes based on degraded or supersaturated depth data

Devices and techniques are generally described for articulated three-dimensional pose tracking. In some examples, a plurality of frames of image data captured by one or more cameras may be received. First feature data representing the plurality of frames of image data may be determined using a backbone network. The first feature data may be projected into three-dimensional (3D) space. In some examples, 3D location data describing respective 3D locations of one or more persons represented by the first feature data projected in the 3D space may be determined. The first feature data and the 3D location data may be sent to a four-dimensional (4D) convolutional neural network (CNN). The 4D CNN may generate second feature data comprising respective 3D representations of the one or more persons. Three dimensional pose data representing articulated 3D pose information for the one or more persons may be generated.

End-to-end multi-person articulated three dimensional pose tracking

Growing community of inventors

Kenmore, WA, United States of America

Jayakrishnan Kumar Eledath