Amazon Technologies, Inc.

Liefeng Bo

Ohil Krishnamurthy Manyam

Gopi Prashanth Gopal

Gerard Guy Medioni

Xiaofeng Ren

Dilip Kumar

Paul Eugene Munger

Yasser Baseer Asmi

Sudarshan Narasimha Raghavan

Jayakrishnan Kumar Eledath

Avishkar Misra

Christopher Robert Towers

Nikhil Chacko

Robert Crandall

Nishitkumar Ashokkumar Desai

Kushagra Srivastava

Ramanathan Palaniappan

Michel Leonard Goldstein

Navid Shiee

Fan Sun

Minmin Chen

Daniel Bibireata

Keunhong Park

Haichao Zhang

Saurabh Sehgal

Described is a system for counting stacked items using image analysis. In one implementation, an image of an inventory location with stacked items is obtained and processed to determine the number of items stacked at the inventory location. In some instances, the item closest to the camera that obtains the image may be the only item viewable in the image. Using image analysis, such as depth mapping or Histogram of Oriented Gradients (HOG) algorithms, the distance of the item from the camera and the shelf of the inventory location can be determined. Using this information, and known dimension information for the item, a count of the number of items stacked at an inventory location may be determined.

Determining inventory changes at an inventory location

Tidiness data associated with an inventory location, such as a shelf, indicates whether an item returned to the inventory location is designated for storage at the inventory location. When an item is returned to the inventory location and a user is determined to be near the inventory location, the weight change detected at the inventory location is compared to a corresponding weight of each item in the custody of the user, such as items within cart state data associated with the user. Differences in weight between the weight change and the corresponding weight of each item in the custody of the user are calculated. Such differences may then be compared to one or more metrics, such as threshold values, to determine how well the top candidate item in the custody of the user matches the type of item designated for storage at the inventory location.

Generating tidiness data associated with items at an inventory location

An event at a shelf, such as a user interacting with items on the shelf, may be detected using sensors associated with the shelf. Hypotheses to describe the event are generated based on the collected sensor data. However, hypotheses determined based on only one type of sensor data may not have a confidence value that is sufficiently reliable. Hypotheses derived from different sensor types may be combined to generate new hypotheses that have higher confidence values. For example, hypotheses based on image data may be combined with hypotheses based on weight data to produce hypotheses with higher confidence values that are sufficiently reliable. A hypothesis, from the set of combined hypotheses, having the highest confidence value may then be used to describe the interaction at the shelf, such as identifying the item and quantity of item the user interacted with during the event.

Sensor data fusion for increased reliability

One or more load cells measure the weight of items at a fixture. Weight changes occur as items are picked from or placed to the fixture and may be used to determine when the item was picked or placed, quantity and so forth. Individual weights for a type of item may vary. A set of data comprising weight changes associated with interactions involving a single one of a particular type of item is gathered. These may be weight changes due to picks, places, or both. A model, such as a probability distribution, may be created that relates a particular weight of that type of item to a probability. The model may then be used to process other weight changes and attempt to determine what type of item was involved in an interaction.

System to determine item weight using data from weight sensors

One or more load cells measure the weight of items on a shelf or other fixture. Weight changes occur as items are picked from or placed to the fixture. Output from the load cells is processed to produce denoised data. The denoised data is processed to determine event data representative of a pick or a place of an item. Hypotheses are generated using information about where particular types of items are stowed, the weights of those particular types of items, and the event data. A high scoring hypothesis is used to determine interaction data indicative of the type and quantity of an item that was added to or removed from the fixture. If ambiguity exists between hypotheses, additional techniques such as data about locations of weight changes and fine grained analysis may be used to determine the interaction data.

Interaction determination using data from weight sensors

A user undertakes an event, such as adding, removing, or otherwise interacting with an item stowed at a fixture. Using successive samples of weight data that occur during an event, a plurality of vectors are generated that are indicative of a weight change and a location associated with the fixture. The vectors are processed to determine where within the fixture the event took place. Hypotheses are generated that describe predicted interactions involving predicted locations that correspond to those indicated by the vectors. The hypotheses are ranked and then one is selected as a solution. The predicted values associated with the selected hypothesis are then used to generate interaction data that indicates one or more types of item and quantities of the items that were added, removed, or otherwise handled by the user.

System to determine user interaction with fixture

Sensor data from load cells at a shelf is processed using a first time window to produce first event data describing coarse and sub-events. Location data is determined that indicates where on the shelf weight changes occurred at particular times. Hypotheses are generated using information about where items are stowed, weights of those of items, type of event, and the location data. If confidence values of these hypotheses are below a threshold value, second event data is determined by merging adjacent sub-events. This second event data is then used to determine second hypotheses which are then assessed. A hypothesis with a high confidence value is used to generate interaction data indicative of picks or places of particular quantities of particular types of items from the shelf.

Joint inference determination of interaction using data from weight sensors

This disclosure describes systems and techniques for detecting certain activity in image data, such as frames of video data. For example, the systems and techniques may create and utilize an activity classifier for detecting and classifying certain human activity in video data of a facility. In some instances, the classifier may be trained to identify, from the video data, certain predefined activity such as a user picking an item from a shelf, a user returning an item to a shelf, a first user passing an item to a second user, or the like. In some instances, the techniques enable activity detection using only video data, rather than in addition to data acquired by other sensors.

Vision-based event detection

This disclosure describes a system for utilizing multiple image processing techniques to identify an item represented in an image. In some implementations, one or more image processing algorithms may be utilized to process a received image to generate item image information and compare the item image information with stored item image information to identify the item. When a similarity score identifying the similarity between the item image information and at least one of the stored item image information is returned, a determination may be made as to whether the similarity score is high enough to confidently identify the item. If it is determined that the similarity score is high enough to confidently identify the item, the other algorithms may be terminated and the determined identity of the item returned.

Tiered processing for item identification

Load cells measure the weight of items on a shelf. Weight changes occur as items are picked from or placed to the fixture. Information about these weight changes is used to determine an estimated location on the shelf of a weight change. Hypotheses are generated using information about where particular types of items are stowed, the weights of those particular types of items, information about the weight changes, and the estimated locations of the weight changes. A model is used to produce confidence values in the hypotheses based on a change in weight measured at a first side and a change in weight measured at a second side of the shelf. A hypothesis with a confidence value that exceeds the threshold may be selected and used to determine interaction data indicative of a quantity picked or placed, type of item, and location on the shelf.

Inferred determination of interaction using data from weight sensors

This disclosure describes techniques for removing noise from a signal to generate a modified signal, with the modified signal preserving any transitions of interest (e.g., sharp-edge discontinuities) present within the initial signal. In one example, the signal comprises a time-series signal with the time series representing a sequence of weight measurements from a scale device. In some examples, the scale device includes a platform that supports one or more physical items that may be selectively removed or added to. Here, the signal may include a sequence of step functions corresponding to changes in weight on the scale device (based on the removal or addition of items on the platform), plus corrupting noise from vibration. The techniques described herein may remove the corrupting noise, while preserving the sharp edges representing sudden changes in weight.

Signal processing techniques

A user may pick an item from a first inventory location, such as in a lane on a shelf, and may return it another location that is assigned to another type of item. Described are techniques to generate tidiness data that is indicative of whether an item has been returned to an inventory location assigned to that type of item. As items are taken, information about the type of item taken and its weight are stored. When an increase in weight at a lane indicates a return of an item to the lane, the weight of the return is compared to the stored weight of the items previously taken by a user. If the weights correspond to within a threshold value, the type of item associated with the stored weight is deemed to be returned and tidiness data indicative of a tidy return of the item to its appointed lane may be generated.

Determination of untidy item return to an inventory location using weight

A user may place an item at an inventory location in an incorrect location, such as the wrong lane on the shelf. An untidy score is generated that is indicative of whether an item is where it is supposed to be. A correct item verification (CIV) score is determined that is indicative of how similar the returned item is to the type of item that is supposed to be stored at that location. An invalid item recognition (IIR) score is determined that is indicative of similarity of the returned item to a set of items associated with the user, that set excluding the item that is supposed to be stored at that location. The CIV and the IIR are combined to generate an untidy score. Based on the untidy score, someone may be dispatched to clean up the inventory location.

Determination of untidy item return to an inventory location

Items may be stowed in an inventory location, such as a shelf. An interaction may take place where one of the items is picked from or placed on the inventory location. Sensor data can be acquired from two or more sensors, such as cameras or weight sensors, configured to collect such sensor data for the inventory location. Hypotheses to describe interactions at the inventory location may be determined using sensor data from each different sensor. However, the confidence value for such hypotheses may not be reliable. Fusion of sensor data may be performed where sets of hypotheses from the different sensors are combined and the resulting hypotheses are more accurate.

Growing community of inventors

Seattle, WA, United States of America

Liefeng Bo