Amazon Technologies, Inc.

Google Inc.

Navid Shiee

Ohil Krishnamurthy Manyam

Ramanathan Palaniappan

Gopi Prashanth Gopal

Michel Leonard Goldstein

Kun Qian

Li Zhang

Aaron Michael Donsbach

Gerard Guy Medioni

Xiaofeng Ren

Liefeng Bo

Arushan Rajasekaram

Christopher Max Breithaupt

Fan Sun

Yasser Baseer Asmi

Alexander Clark Prater

Brett Allen Skaloud

Connor Spencer Blue Worley

Ismael Medrano

Christopher Robert Towers

Ivan Avramov

Jack G Song

Benjamin Lloyd Shapiro

Dilip Kumar

The present disclosure provides systems and methods that provide feedback to a user of an image capture device that includes an artificial intelligence system that analyzes incoming image frames to, for example, determine whether to automatically capture and store the incoming frames. An example system can also, in the viewfinder portion of a user interface presented on a display, a graphical intelligence feedback indicator in association with a live video stream. The graphical intelligence feedback indicator can graphically indicate, for each of a plurality of image frames as such image frame is presented within the viewfinder portion of the user interface, a respective measure of one or more attributes of the respective scene depicted by the image frame output by the artificial intelligence system.

Systems and methods for providing feedback for artificial intelligence-based image capture devices

An event at a shelf, such as a user interacting with items on the shelf, may be detected using sensors associated with the shelf. Hypotheses to describe the event are generated based on the collected sensor data. However, hypotheses determined based on only one type of sensor data may not have a confidence value that is sufficiently reliable. Hypotheses derived from different sensor types may be combined to generate new hypotheses that have higher confidence values. For example, hypotheses based on image data may be combined with hypotheses based on weight data to produce hypotheses with higher confidence values that are sufficiently reliable. A hypothesis, from the set of combined hypotheses, having the highest confidence value may then be used to describe the interaction at the shelf, such as identifying the item and quantity of item the user interacted with during the event.

Sensor data fusion for increased reliability

An inventory location, such as a shelf, may be used to stow different types of items in different areas of the shelf. Interactions may take place, such as the pick or place of items from the shelf. Weight data acquired from weight sensors at the shelf may be used to generate a set of hypotheses to describe types and quantities of items added to or removed from the shelf during an interaction. One or more of these hypotheses may be eliminated from consideration or selected as a solution based on activity data related to the shelf. The activity data is based on non-weight data from a non-weight sensor associated with the shelf. As an example, image data from a camera directed to the shelf can help identify presence of a user or a change in appearance at an area of the shelf to help evaluate hypotheses.

Evaluation of weight hypotheses using non-weight data

Items may be stowed in an inventory location, such as a shelf. An interaction may take place where one of the items is picked from or placed on the inventory location. Sensor data can be acquired from two or more sensors, such as cameras or weight sensors, configured to collect such sensor data for the inventory location. Hypotheses to describe interactions at the inventory location may be determined using sensor data from each different sensor. However, the confidence value for such hypotheses may not be reliable. Fusion of sensor data may be performed where sets of hypotheses from the different sensors are combined and the resulting hypotheses are more accurate.

Fusion of sensor data for detecting interactions at an inventory location

An item stowed at an inventory location may rest or be supported by a surface, such as a shelf, that includes an optical sensor array. The optical sensor array comprises a plurality of optical sensors that detect light intensity values. When a light source illuminates the item resting on the surface, a footprint or shadow is cast by the item onto the surface. Such a shadow is detected by the light intensity values at the optical sensors in the optical sensor array. Once the shadow is detected, information about the area, shape, and perimeter of the shadow can be determined and saved as footprint data for a single item. Such a process can be performed at intake of the item to the facility and later accessed to determine a quantity of the item on a shelf of a facility.

Generation of footprint data for items

An inventory location such as a shelf may be used to stow different types of items. Interactions may take place, such as the pick or place of one or more items from the inventory location. Weight data may be acquired from weight sensors coupled to the shelf. A measured change in weight is used to determine a first group of hypotheses. Each of the hypotheses may be descriptive of a different permutation of products being added to or removed from the shelf that could account for the measured change in weight. The weight data is also processed to determine a weight distribution of the items on the shelf. Information about the weight distribution may be used to select a subset of the first group of hypotheses. The subset may be used to determine what items were interacted with, quantity involved, and so forth.

Item determination based on weight data

Described are systems for processing sensor data associated with an inventory location for the purpose of identifying stable and unstable states in the sensor data. The stable and unstable states can be determined by fitting a line or curve to a portion of the sensor data occurring during a window of time and calculating a corresponding slope value of the line or curve. By comparing the slope value of the line or curve to a threshold value, stable and unstable states can be determined for the sensor data. Once stable states have been identified, a change in sensor data can be determined and used to identify a change in quantity of items at the inventory location.

Sensor data processing for detection of changes in item quantities

An inventory location, such as a shelf, may be used to stow different items. The shelf is divided into a plurality of adjacent lanes, some of which may be grouped into a cluster. Items are placed upon the lanes of the cluster. Each lane includes one or more weight sensors to obtain weight data about a load on the lane. Based on the weight data and physical layout data for the shelf, one or more hypotheses may be determined for describing interactions, such as a pick or place of an item, occurring at the lanes of a cluster. One of the hypotheses may be selected as a solution for describing the interaction and used to maintain inventory quantities of items at the shelf.

Inventory management using weight sensors and physical layout data

Described are systems for processing sensor data obtained from a sensor such as a weight sensor. In one implementation, a line may be fit to a set of sensor data occurring during a window of time. A slope value of the line may be calculated and provided as input to a change detector. The change detector may implement a cumulative sum (CUSUM) algorithm to determine when the slope value is indicative of a stable or unstable state. Values of sensor data associated with times indicated as having a stable state may be used to determine weight data.

Sensor data processing system

An item stowed at an inventory location may rest on a surface such as a shelf or may be supported above the surface. The item may exhibit a footprint that is representative of a physical perimeter of the item. For example, the footprint may be an edge of a shadow cast by the item. Information about a size and shape of a footprint associated with a particular item may be stored. An optical sensor array may acquire sensor image data indicative of the footprint of one or more of the items stored at an inventory location. A quantity of items at the inventory location may be determined by comparing the sensor image data with the stored data.

Optical item management system

An inventory location such as a shelf may be used to stow different types of items, with each type of item in a different partitioned area or section of the shelf. Weight data from weight sensors coupled to the shelf is used to determine a change in weight of the shelf and a change in the center-of-mass ('COM') of the items on the shelf. Based on the weight data and item data indicative of what items are stowed in particular partitioned areas, activity such as a pick or place of an item and the partitioned area in which the activity occurred may be determined. Data from other sensors, such as a camera, may be used to confirm the occurrence of the activity, disambiguate the determination of the particular partitioned area, and so forth.

Item management system using weight

An inventory location such as a shelf may be used to stow different types of items. Interactions may take place, such as the pick or place of one or more items from the inventory location. Image data may be acquired from cameras viewing the shelf and weight data may be acquired from weight sensors coupled to the shelf. Hypotheses may be determined that indicate possible interactions with the inventory location, such as pick or place of an item with regard to the inventory location, and the probability that those interactions are correct. The hypotheses and their associated probabilities may be aggregated. From the aggregated hypotheses, a hypothesis with a highest confidence value may be deemed a solution.

Multiple sensor data fusion system

An inventory location such as a shelf may be used to stow different types of items. The shelf may be equipped with a plurality of lanes arranged parallel to one another. Items may rest upon two or more of the lanes. Each lane includes a weight sensor that provides weight data about a load on the lane. Based on the weight data and item data indicative of individual weights of those items stowed on the shelf, interaction data indicative of an activity such as a pick or place of an item may be determined. The interaction data may specify what item was picked or placed, quantity of that item that was picked or placed, and so forth. Data from other sensors, such as a camera, may be used to determine the interaction data.

Growing community of inventors

Seattle, WA, United States of America

Navid Shiee