Amazon Technologies, Inc.

Nikhil Chacko

Gerard Guy Medioni

Nishitkumar Ashokkumar Desai

Jayakrishnan Kumar Eledath

Liefeng Bo

Dilip Kumar

Paul Eugene Munger

Robert Crandall

Gopi Prashanth Gopal

Kushagra Srivastava

Pahal Kamlesh Dalal

Alessandro Bergamo

Kaustav Kundu

Keunhong Park

Marian Nasr Amin George

Keshav Nand Tripathi

Jingjing Liu

This disclosure describes systems and techniques for identifying events that occur within an environment using image data captured at the environment. For example, one or more cameras may generate image data representative of a user interacting with an item on the shelf. This image data may be used to generate feature data associated with the user and the item, which may be analyzed by one or more classifiers for identifying an interaction between the user and the item. The systems and techniques may then generate interaction data, which in turn may be analyzed by one or more additional classifiers for identifying an event, such as the user picking a particular item from the shelf within the environment. Event data indicative of the event may then be used to update a virtual cart of the user.

System and method for vision-based event detection

One or more load cells measure the weight of items at a fixture. Weight changes occur as items are picked from or placed to the fixture and may be used to determine when the item was picked or placed, quantity and so forth. Individual weights for a type of item may vary. A set of data comprising weight changes associated with interactions involving a single one of a particular type of item is gathered. These may be weight changes due to picks, places, or both. A model, such as a probability distribution, may be created that relates a particular weight of that type of item to a probability. The model may then be used to process other weight changes and attempt to determine what type of item was involved in an interaction.

System to determine item weight using data from weight sensors

One or more load cells measure the weight of items on a shelf or other fixture. Weight changes occur as items are picked from or placed to the fixture. Output from the load cells is processed to produce denoised data. The denoised data is processed to determine event data representative of a pick or a place of an item. Hypotheses are generated using information about where particular types of items are stowed, the weights of those particular types of items, and the event data. A high scoring hypothesis is used to determine interaction data indicative of the type and quantity of an item that was added to or removed from the fixture. If ambiguity exists between hypotheses, additional techniques such as data about locations of weight changes and fine grained analysis may be used to determine the interaction data.

Interaction determination using data from weight sensors

A user undertakes an event, such as adding, removing, or otherwise interacting with an item stowed at a fixture. Using successive samples of weight data that occur during an event, a plurality of vectors are generated that are indicative of a weight change and a location associated with the fixture. The vectors are processed to determine where within the fixture the event took place. Hypotheses are generated that describe predicted interactions involving predicted locations that correspond to those indicated by the vectors. The hypotheses are ranked and then one is selected as a solution. The predicted values associated with the selected hypothesis are then used to generate interaction data that indicates one or more types of item and quantities of the items that were added, removed, or otherwise handled by the user.

System to determine user interaction with fixture

Sensor data from load cells at a shelf is processed using a first time window to produce first event data describing coarse and sub-events. Location data is determined that indicates where on the shelf weight changes occurred at particular times. Hypotheses are generated using information about where items are stowed, weights of those of items, type of event, and the location data. If confidence values of these hypotheses are below a threshold value, second event data is determined by merging adjacent sub-events. This second event data is then used to determine second hypotheses which are then assessed. A hypothesis with a high confidence value is used to generate interaction data indicative of picks or places of particular quantities of particular types of items from the shelf.

Joint inference determination of interaction using data from weight sensors

Sensors in a facility obtain sensor data about a user's interaction with a fixture, such as a shelf. The sensor data may include images such as obtained from overhead cameras, weight changes from weight sensors at the shelf, and so forth. Based on the sensor data, one or more hypotheses that indicate the items and quantity may be determined and assessed. The hypotheses may be based on information such as the location of a user and where their hands are, weight changes, physical layout data indicative of where items are stowed, cart data indicative of what items are in the possession of the user, and so forth. A hypothesis having a greatest confidence value may be deemed to be representative of the user's interaction, and interaction data indicative of the item and quantity may be stored.

This disclosure describes systems and techniques for detecting certain activity in image data, such as frames of video data. For example, the systems and techniques may create and utilize an activity classifier for detecting and classifying certain human activity in video data of a facility. In some instances, the classifier may be trained to identify, from the video data, certain predefined activity such as a user picking an item from a shelf, a user returning an item to a shelf, a first user passing an item to a second user, or the like. In some instances, the techniques enable activity detection using only video data, rather than in addition to data acquired by other sensors.

Vision-based event detection

Load cells measure the weight of items on a shelf. Weight changes occur as items are picked from or placed to the fixture. Information about these weight changes is used to determine an estimated location on the shelf of a weight change. Hypotheses are generated using information about where particular types of items are stowed, the weights of those particular types of items, information about the weight changes, and the estimated locations of the weight changes. A model is used to produce confidence values in the hypotheses based on a change in weight measured at a first side and a change in weight measured at a second side of the shelf. A hypothesis with a confidence value that exceeds the threshold may be selected and used to determine interaction data indicative of a quantity picked or placed, type of item, and location on the shelf.

Inferred determination of interaction using data from weight sensors

This disclosure describes techniques for removing noise from a signal to generate a modified signal, with the modified signal preserving any transitions of interest (e.g., sharp-edge discontinuities) present within the initial signal. In one example, the signal comprises a time-series signal with the time series representing a sequence of weight measurements from a scale device. In some examples, the scale device includes a platform that supports one or more physical items that may be selectively removed or added to. Here, the signal may include a sequence of step functions corresponding to changes in weight on the scale device (based on the removal or addition of items on the platform), plus corrupting noise from vibration. The techniques described herein may remove the corrupting noise, while preserving the sharp edges representing sudden changes in weight.

Growing community of inventors

Seattle, WA, United States of America

Nikhil Chacko