Amazon Technologies, Inc.

Sensormatic Electronics Corporation

Mirko Ristivojevic

Jeremy Samuel De Bonet

Oded Maron

Ejaz Ahmed

Ian Christopher Westmacott

Changchun Li

Matthew A Gruenke

Kilho Son

Jonathan Evan Cohn

Daniel Ron

This disclosure is directed to, in part, a processing pipeline for detecting predefined activity using image data, identifying an item-of-interest and a location of the item-of-interest across frames of the image data, determining a trajectory of the item-of-interest, and determining an identifier of the item-of-interest and an action taken with respect to the item-of-interest. The processing pipeline may utilize one or more trained classifiers and, in some instances, additional data to identify items that are placed into or removed from a tote (e.g., basket, cart, or other receptacle) by users in material handling facilities as the users move around the material handling facilities.

Using sensor data to determine activity

This disclosure is directed to using cascading algorithms to automatically identify items placed in a tote or other receptacle utilized by users in material handling facilities as the users move around the facilities. A tote may store a database or 'gallery' of item representations for all of the items that are stored in the facility that a user may place in their totes. The tote may use multiple algorithms in a cascading manner to analyze the gallery of item representations in order to iteratively narrow the search space of item representations in the gallery to determine which of the items was placed in the tote by a user. Upon identifying the item placed in the tote, the tote may add an item identifier for the item to a virtual listing of item identifiers representing items previously placed in the tote.

Identifying items using cascading algorithms

A tag with sufficient variability is generated. The tag facilitates quick location and processing within a facility. During generation, a matrix representative of a possible tag may be tested for sufficient variability. A two-dimensional matrix representing the possible tag may include different bit values at different positions. A variability metric of the matrix is performed to check the variability between adjacent elements of the matrix. This variability metric of the matrix is then compared against a variability threshold value. If the variability metric of the matrix falls below the threshold value, the matrix is flagged as unusable. If the variability metric of the matrix exceeds the threshold value, the matrix can be used to generate a tag with sufficient variability. Once generated, the tag can be presented and affixed to an item or object to help identify and locate the item within the facility.

System for generating tags

A tag comprises cells represented by marks, such as white and black, arranged in a predetermined pattern. Reference cells are used to determine the edges of the tag and orientation, while others are used to encode data and error correction data. During processing, a tag is detected based on a variability metric within a given region of an image exceeding a threshold value. Once detected, the orientation of the tag is determined using the reference cells. Confidence values, indicative of a likelihood that the cell is a particular cell value, such as white or black, are determined for each cell in a first set of cell values. If the first set is invalid, the cell values for a set of lowest confidence cell values are inverted to produce a second set of cell values. The second set is tested, and if valid, is used to produce output data.

Error tolerant system for reading tags

Tags comprising marks arranged in a predetermined pattern are applied to an item. The marks are made with an ink that fluoresces under infrared (IR) light. A camera with a filter acquires an image of the light emitted by the fluorescence of the ink. This image is processed to determine a portion of the image in which the tag is located. Once located, that portion is processed to rectify and align the tag. This rectified image is then processed to read out tag data that is encoded by the arrangement of marks. The tag data may then be used to identify the item, designate shipping information, and so forth.

System for reading tags

A method and system for analyzing video data in a security system. An analysis compares a current frame to a background model. The analysis system compares the background model to the current frame to identify changed pixel patches. The analysis system uses morphological image processing to generate masks based on the changed pixel patches. Next, the analysis system applies the masks to the background model and the current frames to determine whether the changed pixel patches are characteristic of abandoned or removed objects within the video data.

Method and system to characterize video background changes as abandoned or removed objects

A system and method for determining proximity of individuals to specific regions of interest such as Point of Sale ('POS') terminals within a POS monitored area of a scene of video data captured from security cameras as part of a networked security system, when the security cameras are located outside the POS monitored area, mounted on a ceiling or other high location and pointed at the POS monitored area. With the help of customer regions drawn in an abstract layer whose areas coincide with expected locations of individuals near POS terminals in the video data, and video analytics elements such as bounding boxes generated around individuals in the video data, the system can perform live and forensic analysis of the video data to infer information such as the proximity of individuals to POS terminals and the relative height of an individual compared to their expected height within the scene of video data.

Growing community of inventors

Cambridge, MA, United States of America

Mirko Ristivojevic