Target Brands, Inc.

Todd A Hagen

Andrew James Wipf

Donnie Scott Tolbert

Arne Wilkin

Josh Michael Dingman

Donald John Armstrong

Daniel Woods

Tomas Kadlec

Nicholas Lojewski

John Ojanen

Yeshwanth Gowda

Christopher Mark Brakob

Yeshwanth M Gowda

Donnie Tolbert

Disclosed are systems and techniques for determining out of stock conditions on shelves. The techniques can include receiving, by a computing system, image data from a camera having pixel locations that each uniquely address and store a pixel value, generating a backing map having cell locations that each uniquely address and share a unique address with a corresponding pixel location in the image data, each cell location storing a backing value being an empty value if the pixel value is classified as showing the backing of a shelf and the backing value being a nonempty value if the pixel value is classified as not showing the backing of the shelf, determining, in the backing map, a shelf area representing a location of the captured shelf, and identifying an empty area by finding an area above the shelf area containing a threshold number of cell locations storing the empty value.

Mobile apparatus with computer vision elements for classifying shelf-space

Described herein are systems and techniques for imaging inventory objects in an environment. A system can include a cart, a first fixed camera fixedly mounted on the cart at a first angle, a pan-tilt-zoom (PTZ) camera controllably mounted on the cart, a PTZ controller, and a cart controller. The PTZ controller can receive PTZ instructions from the cart controller and send engagement instructions to the PTZ camera. The cart controller can receive, from the first fixed camera, first image data that captures a first inventory object, determine, from the first image data, a spatial location of a first inventory object, generate PTZ instructions to cause the PTZ camera to capture the first inventory object, transmit the PTZ instructions to the PTZ controller, and receive PTZ image data that captures the first inventory object.

Mobile apparatus with computer vision elements for inventory condition

The disclosed technology provides for reconstructing products that are added to a shopping cart in three-dimensional space. The reconstructed products can be used for product validation. A shopping cart can include product validation hardware including light emitters and light sensors along a top perimeter of the shopping cart. The shopping cart can activate the light emitters to emit light across a top horizontal plane of the shopping cart, receive, from light sensors on a side of the shopping cart opposite a side where the activated light emitter is located, light intensity data as a product passes through the top horizontal plane and obstructs at least a portion of the emitted light from being detected by the light sensors, identify, from the light intensity data, slices of the product, and reconstruct the product in 3D space based on stitching together the slices.

Shopping cart with infrared product validation

Described herein are systems and methods for validating products that are added to a shopping cart. A shopping cart that can include product validation hardware and a controller, the product validation hardware including motion sensors positioned at least along a top perimeter of the shopping cart and at least one weight sensor positioned inside the shopping cart. The shopping cart can receive, from one or more of the motion sensors, motion data when the product is detected to be entering a top horizontal plane of the shopping cart, receive, from the at least one weight sensor, weight change data that is collected in real-time as the product settles inside the shopping cart, and correlate the weight change data with the motion data to generate a weight signature over time for the product. The weight signature can be used to validate the product that is added to the shopping cart.

Shopping cart with weight bump validation

Physical shopping carts can have product detection systems and associate physical shopping carts with mobile computing devices (e.g., smartphones, tablet computing devices, smart watches, wearable computing devices). For example, physical shopping carts can be equipped with one or more product detection systems (e.g., scanners, sensors, cameras) that can electronically tally products that are placed in physical shopping carts. Mobile computing devices can be associated with and mounted on the physical shopping carts to provide a variety of enhanced shopping cart features not possible with conventional physical shopping carts, such as electronically tracking the contents of a shopping cart, checking-out from the mobile computing device (instead of at conventional check-out areas, such as point of sale terminals), and others.

Physical shopping cart having features for use in customer checkout of items placed into the shopping cart

In some implementations, a system for verifying items in a retail environment includes a physical shopping cart including a first set of sensors, and an automated checkout station including a second, different set of sensors. The physical shopping cart receives item verification data for verifying an item, detects the item as it enters the physical shopping cart, and performs a primary verification of the item. The automated checkout station obtains a virtual shopping cart that corresponds to the physical shopping cart. The virtual shopping cart includes a list of items that have been placed in the physical shopping cart, and a verification status of each item. The second, different set of sensors generate station sensor data that represents the physical shopping cart and the items in the physical shopping cart. A secondary verification of the physical shopping cart and its contents is performed by the automated checkout station.

Verification of items by automated checkout system

In some implementations, a system for validating products that are placed in a shopping cart includes a shopping cart configured to retain products and to be operated by a user in a retail environment. The shopping cart includes product validation hardware and a controller. The product validation hardware includes motion sensors positioned along a top perimeter of the shopping cart and at least one sound sensor positioned inside the shopping cart. The shopping cart is configured to receive, from the motion sensors, motion data when a product enters a top horizontal plane of the shopping cart, and to receive, from the sound sensor, audio data that is collected in real-time as the product is placed in the shopping cart. An audio signature is generated for the product, and is used to validate the product that is placed in the shopping cart.

Shopping cart with sound-based validation

In some implementations, a method performed by data processing apparatuses includes receiving, by an item verification engine, a scan notification indicating that a mobile computing device has scanned an item. In response to receiving the scan notification, the item verification engine provides an instruction for a cart computing system to activate item sensors on a shopping cart. The item verification engine receives sensor data collected by the item sensors as a result of the item having been placed in the shopping cart. The item verification engine receives item verification data for verifying the scanned item from the mobile computing device, performs a verification of the item based on the sensor data and the item verification data, and provides verification results for presentation by the mobile computing device.

Verification of items placed in physical shopping cart

Disclosed are systems and techniques for identifying product identifiers in images. A technique can include receiving, by an edge computing device, x and y coordinates for a location of an out of stock shelf section, determining a frame of reference (FOR) based on the coordinates, the FOR including a location of a product identifier for the out of stock shelf section, determining incremental adjustments to a camera based on the FOR, instructing the camera to adjust by the incremental adjustments and capture an image of the location of the product identifier, performing image analysis on the image to identify the product identifier, determining second incremental adjustments to the camera to focus the camera on the location of the product identifier if the product identifier is not identifiable in the image, and instructing the camera to reset to original camera settings if the product identifier is identifiable in the image.

Mobile apparatus with computer vision elements for product identifier detection with minimal detection adjustments

In some implementations, a system for providing and managing associations between mobile computing device and physical shopping carts used in retail stores includes a physical shopping cart configured to transport physical goods around a store. The physical shopping cart includes a product detection system that is programmed to detect products that are placed in the physical shopping cart, a wireless transceiver to wirelessly communicate with other devices, and a cart identifier that uniquely identifies the physical shopping cart. The system further includes a mobile computing device that is programmed (i) to obtain the cart identifier for the shopping cart, (ii) to establish an association with the physical shopping cart using the cart identifier, and (iii) once the association between the physical shopping cart and the mobile computing device has been established, to output a user interface that provides a real-time summary of the products in the physical shopping cart.

Physical shopping chart-to-mobile device associations

A method includes monitoring performance of a system that determines locations of items in a retail store based on sensor values from a collection of sensors in the retail store. When the performance is insufficient to process the sensor values as the sensor values are received, a spatial resolution used to determine the locations of the items in the retail store is reduced.

Growing community of inventors

Minneapolis, MN, United States of America

Todd A Hagen