Amazon Technologies, Inc.

Alessandro Bergamo

Nishitkumar Ashokkumar Desai

Jayakrishnan Kumar Eledath

Pahal Kamlesh Dalal

Marian Nasr Amin George

Gerard Guy Medioni

Joseph Patrick Tighe

Jean Laurent Guigues

Vivek Bhadauria

Robert Matthias Steele

Kartik Muktinutalapati

Shubham Chandra Gupta

Lu Xia

Prathyusha Reddy Cheruku

Nikhil Chacko

Kaustav Kundu

Keshav Nand Tripathi

Jingjing Liu

Techniques for unique label recognition and notification for streaming video. A video stream from a camera-equipped electronic device is received via the API endpoint of a stream processing service in a provider network. A request for unique label recognition and notification is received via an API endpoint of a computer vision service in the provider network. The computer vision service analyzes a video fragment of the video stream. If a unique label is recognized in the video fragment, then a notification service in the provider network sends a notification that the unique label was recognized in video captured by the camera-equipped electronic device.

Unique label recognition and notification for streaming video

Techniques for label recognition and notification for streaming video from non-overlapping cameras. A stream processing service of a provider network receives a first video stream from a first camera-equipped electronic device via an API endpoint of the stream processing service. The stream processing service also receives a second video stream from a second camera-equipped electronic device an API endpoint of the stream processing service. Meanwhile, a request for label recognition and notification is received at a computer vision service of the provider network via an API endpoint of the computer vision service. In response, the computer vision service recognizes a label in a video fragment of the first camera video stream and recognizes a label in a video fragment of the second camera video stream, and then identifies whether the two labels are the same label. If so, a notification service of the provider network sends a notification indicating that the label was recognized across non-overlapping cameras.

Label recognition and notification for streaming video from non-overlapping cameras

This disclosure describes systems and techniques for identifying events that occur within an environment using image data captured at the environment. For example, one or more cameras may generate image data representative of a user interacting with an item on the shelf. This image data may be used to generate feature data associated with the user and the item, which may be analyzed by one or more classifiers for identifying an interaction between the user and the item. The systems and techniques may then generate interaction data, which in turn may be analyzed by one or more additional classifiers for identifying an event, such as the user picking a particular item from the shelf within the environment. Event data indicative of the event may then be used to update a virtual cart of the user.

System and method for vision-based event detection

A camera acquires an image of an item involved in an interaction, such as the item being picked or placed on a shelf by a user. The item depicted in the image is identified via item recognition using a customized neural network that uses previously trained subnetworks for a set of candidate items. The candidate items are determined based on items near a user location, items associated with a user cart, items in a shopping list or wish list associated with the user, or items otherwise associated with the user. Each previously trained subnetwork for a candidate item includes one or more reference images and corresponding reference feature data. The customized neural network uses a matching and inference network to test the image against the candidate items and provide comparison data indicative of a similarity between the item depicted in the image and one of the candidate items.

Customized neural network for item recognition

In a materials handling facility, events may be associated with users based on imaging data captured from multiple fields of view. When an event is detected at a location within the fields of view of multiple cameras, two or more of the cameras may be identified as having captured images of the location at a time of the event. Users within the materials handling facility may be identified from images captured prior to, during or after the event, and visual representations of the respective actors may be generated from the images. The event may be associated with one of the users based on distances between the users' hands and the location of the event, as determined from the visual representations, or based on imaging data captured from the users' hands, which may be processed to determine which, if any, of such hands includes an item associated with the event.

Associating items with actors based on digital imagery

A user interacts with an inventory location, such as a lane on a shelf, picking items or placing items to that inventory location. A neural network (NN) is trained using a subset of reference images to determine parameters, such as node weights. An image of the user interaction is acquired and processed by the NN to determine if an item in a test image matches an item in a candidate set of items. During use, the parameters are used to dynamically instantiate an ad hoc neural network having a graph that is based on a number of items in a candidate set. The candidate set of items may comprise items within reach of a user at the time the test image was acquired, such as on a nearby shelf or tote. The NN output indicates if the test image is deemed to match one of the candidate set of items.

Item recognition system using reference images

Associating events with actors based on digital imagery

Growing community of inventors

Seattle, WA, United States of America

Alessandro Bergamo