Amazon Technologies, Inc.

Alibaba Group Holding Limited

Alibaba (china) Co., Ltd.

Xiaofeng Ren

Ohil Krishnamurthy Manyam

Yasser Baseer Asmi

Sudarshan Narasimha Raghavan

Gopi Prashanth Gopal

Liefeng Bo

Christopher Robert Towers

Dilip Kumar

Avishkar Misra

Michel Leonard Goldstein

Sridhar Boyapati

Daniel Bibireata

Ramanathan Palaniappan

Chang Yuan

Alexander Clark Prater

Navid Shiee

Jaechul Kim

Elie Micah Kornfield

Minmin Chen

Nishitkumar Ashokkumar Desai

Emilio Ian Maldonado

Dong Nie

Meng Wang

Wei Xia

Ranju Das

Frank Florian Liberato, Jr

Chen Xu

Jia Xue

Described is a system for counting stacked items using image analysis. In one implementation, an image of an inventory location with stacked items is obtained and processed to determine the number of items stacked at the inventory location. In some instances, the item closest to the camera that obtains the image may be the only item viewable in the image. Using image analysis, such as depth mapping or Histogram of Oriented Gradients (HOG) algorithms, the distance of the item from the camera and the shelf of the inventory location can be determined. Using this information, and known dimension information for the item, a count of the number of items stacked at an inventory location may be determined.

Determining inventory changes at an inventory location

This disclosure describes a system for tracking removal or placement of items at inventory locations with a materials handling facility. In some instances, a user may remove an item from an inventory location and the inventory management system may detect that removal and update a user item list associated with the user to include an item identifier representative of the removed item. Likewise, if the user places an item at an inventory location, the inventory management system may detect that placement and update the user item list to remove an item identifier representative of the placed item.

Detecting inventory changes

This application describes cross-scale point cloud segmentation network architecture for and exemplary systems that utilize such network architecture for semantic segmentation of a point cloud. An embodiment of the network architecture includes an encoding path comprising a plurality of sequentially connected encoding nodes, a decoding path following the encoding path and comprising a plurality of sequentially connected decoding nodes, and a plurality of data links respectively corresponding to a plurality of levels of feature resolution, in which each of the plurality of data links connects one of the plurality of encoding nodes and one of the plurality of decoding nodes that have a same level of feature resolution.

Pyramid architecture for multi-scale processing in point cloud segmentation

A learning model may provide a hierarchy of convolutional layers configured to perform convolutions upon image features, each layer other than a topmost layer convoluting the image features at a lower resolution to a higher layer, and each layer other than a bottommost layer returning the image features to a lower layer. Each layer fuses the lower resolution image features received from a higher layer with same resolution image features convoluted at the layer, so as to combine large-scale and small-scale features of images. Layers of the hierarchy may be substantially equal to a number of lateral convolutions at a bottommost convolutional layer. The bottommost convolutional layer ultimately passes the fused features to an attention mapping module, which utilizes two attention mapping pathways in combination to detect non-local dependencies and interactions between large-scale and small-scale features of images without de-emphasizing local interactions.

Learning model architecture for image data semantic segmentation

This disclosure describes a system for automatically identifying an item from among a variation of items of a same type. For example, an image may be processed and resulting item image information compared with stored item image information to determine a type of item represented in the image. If the matching stored item image information is part of a cluster, the item image information may then be compared with distinctive features associated with stored item image information of the cluster to determine the variation of the item represented in the received image.

Item identification among a variant of items

Sensors in a facility generate sensor data associated with a region of the facility, which can be used to determine a 3D location of an object in the facility. Some sensors may sense overlapping regions of the facility. For example, a first sensor may generate data associated with a first region of the facility, while a second sensor may generate data associated with a second region of the facility that partially overlaps the first region. Sensors may fail at times as determined from sensor output data or status data. In response to identifying a failed sensor, an undetected region corresponding to the failed sensor is identified, as well as a substitute sensor that partially senses the undetected region. Sensor data from the substitute sensor, such as 2D data, is acquired and used to estimate a 3D location of an object in the undetected region.

Management of sensor failure in a facility

This disclosure describes a system for utilizing multiple image processing techniques to identify an item represented in an image. In some implementations, one or more image processing algorithms may be utilized to process a received image to generate item image information and compare the item image information with stored item image information to identify the item. When a similarity score identifying the similarity between the item image information and at least one of the stored item image information is returned, a determination may be made as to whether the similarity score is high enough to confidently identify the item. If it is determined that the similarity score is high enough to confidently identify the item, the other algorithms may be terminated and the determined identity of the item returned.

Tiered processing for item identification

People represented in multiple images can be recognized using accurate facial similarity metrics, where the accuracy can be further improved using contextual information. A set of models can be trained to process image data, and facial features can be extracted from a face region of an image and passed to the trained models. Resulting feature vectors can be concatenated and the dimensionality reduced to generate a highly accurate feature vector that is representative of the face in the image. The feature vector can be used to locate similar vectors in a multi-dimensional vector space, where similarity can be determined based at least in part upon the distance between the endpoints of those vectors in the vector space. Context information from the image can be used to adjust the similarity determination. Similar vectors can be clustered together such that the faces represented by those images are associated with the same person.

Context-inclusive face clustering

An item stowed at an inventory location may rest or be supported by a surface, such as a shelf, that includes an optical sensor array. The optical sensor array comprises a plurality of optical sensors that detect light intensity values. When a light source illuminates the item resting on the surface, a footprint or shadow is cast by the item onto the surface. Such a shadow is detected by the light intensity values at the optical sensors in the optical sensor array. Once the shadow is detected, information about the area, shape, and perimeter of the shadow can be determined and saved as footprint data for a single item. Such a process can be performed at intake of the item to the facility and later accessed to determine a quantity of the item on a shelf of a facility.

Generation of footprint data for items

During operation, a facility may utilize many sensors, such as cameras, to generate sensor data. The sensor data may be processed by an inventory management system to track objects, determine the occurrence of events at the facility, and so forth. At any given time, some of these sensors may fail to provide timely data, may fail to provide any data, may generate inaccurate data, and so forth. Described are techniques to determine failure of sensors and adjust operation of the inventory management system to maintain operability during sensor failure.

Image based inventory item counting

Item variation management

This disclosure describes a system for processing an image of an item and correctly identifying the item from a group of candidate items. In one implementation, as item image information for a new item is added to an item images data store, a determination is made as to the weight of the item represented by the image, and the item may be associated with a weight class. Each weight class represents items within a defined weight range. Item image information for items in the same weight class may then be used when new items are added to inventory and/or when identifying an item represented in an image.

Growing community of inventors

Yarrow Point, WA, United States of America

Xiaofeng Ren