Accel Robotics Corporation

Brain Corporation

Qualcomm Technologies, Inc.

Marius Buibas

John Quinn

Michael Brandon Maseda

Martin Alan Cseh

Csaba Petre

Kaylee Feigum

Vadim Polonichko

Eugene Izhikevich

Micah Richert

Aleksander Bapst

Botond Szatmary

Jose Miguel Rodriguez

Nicholas J Morozovsky

Mark Graham

Chin-Chang Kuo

Jeffrey Alexander Levin

Filip Lukasz Piekniewski

Charles Wheeler Sweet, Iii

Dimitry Fisher

Mark Stirling Caskey

Christopher Lai

Tanuj Pankaj

Yohei Yamamuro

Samir Singh

Matthew Walker

Soheyl Yousefisahi

System that facilitates rapid onboarding of an autonomous (cashier-less) store by capturing images of the store's items from multiple angles, with varying background colors, and that builds a classifier training dataset from these images. Background surfaces may for example be coated with retroreflective tape or film, and variable-color incident light sources may generate the desired background colors. Embodiments may automatically rotate or otherwise reorient the item placed in the onboarding system, so that a relatively small number of cameras can capture views from multiple angles. When an item is placed in the system, a fully automated process may generate a sequence of item orientations and background colors, and may capture and process images from the cameras to create training images. Images of the item from multiple angles, under varying lighting conditions, may be captured without requiring an operator to move or reorient the item.

Multi-lighting conditions rapid onboarding system for visual item classification

System that allows devices of an autonomous store, such as sensors in product display areas, to receive power and communicate data over conductive rails of store fixtures. By using fixtures to transmit data and power, the need for cabling to these devices, or for batteries to power the devices, is eliminated or greatly reduced, thereby dramatically simplifying installation and maintenance. Illustrative fixtures over which devices can communicate include slatwalls, pegboards, and rectangular support bars. Power and data may be multiplexed onto the same pair of conductive rails. Embodiments may use device hubs to coordinate communication with devices and to act as gateways between devices and centralized store servers. Devices may communicate their identities and locations to store servers to facilitate installation; for example, they may display their identities on electronic labels that are imaged by store cameras, so that the store server can learn the location of each device automatically.

Store device network that transmits power and data through mounting fixtures

System that facilitates rapid onboarding of an autonomous (cashier-less) store by capturing images of the store's items from multiple angles, with varying backgrounds, and that builds a classifier training dataset from these images. The system may have cameras in different positions, and backgrounds that generate different background colors. It may have a platform for items that can be switched between a transparent and non-transparent state; cameras below the platform may therefore capture images of the bottom side of the item when the platform is transparent. When an item is placed in the imaging system, a fully automated process may generate a sequence of background colors and may capture and process images from all of the cameras to create training images. Images of the item from multiple angles, including views of the entire external surface of the item, may be captured without requiring an operator to move or reorient the item.

Multi-angle rapid onboarding system for visual item classification

A projected image item tracking system that analyzes projected camera images to determine items taken from, placed on, or moved on a shelf or other area in an autonomous store. The items and actions performed on them may then be attributed to a shopper near the area. Projected images may be combined to generate a 3D volume difference between the state of the area before and after shopper interaction. The volume difference may be calculated using plane-sweep stereo, or using convolutional neural networks. Because these methods may be computationally intensive, the system may first localize a change volume where items appear to have been displaced, and then generate a volume difference only within that change volume. This optimization results in significant savings in power consumption and in more rapid identification of items. The 3D volume difference may also indicate the quantity of items displaced, for example from a vertical stack.

Multi-surface image projection item tracking system

A system having a bar containing distance sensors, such as LIDARs, that may be installed behind a shelf in an automated store. Shelves may be divided into storage zones, such as bins or lanes, and a distance sensor may measure the item quantity in each zone. Quantity changes indicate that a shopper has taken or placed items in the zone. Quantity changes may trigger analysis of camera images of the shelf to identify the items taken or replaced. The distance sensor elements within the bar may be relocatable so they can be positioned correctly behind the corresponding storage zones. The bar may have mounting mechanisms on either side for attachment to shelving support structures. These mounting mechanisms may include security locks. The bar may rotate relative to the mounting mechanisms to provide access to the shelf from behind.

System having a bar of relocatable distance sensors that detect stock changes in a storage area

An autonomous store that tracks shopper movements and actions, and performs store cleaning actions based on analysis of shopper activity. Cleaning actions may include disinfecting the store or regions within the store using radiation, fogging or spraying, or ventilation. Cleaning actions may be targeted; for example, zones where shoppers linger or congregate may be cleaned more frequently or intensively, or shelves or items that shoppers touch may be cleaned after these interactions. Shopper activity information may be used to limit the number of shoppers in a store at once, for example by denying entry when the store is at capacity. The density of shoppers in regions of the store may be communicated to shoppers so that they can limit their interactions with other shoppers. Shopper activity history may be used for contact tracing by identifying other shoppers that may have been exposed to an individual who was in the store.

Self-cleaning autonomous store

Person and projected image item tracking system

Data streams from multiple image sensors may be combined in order to form, for example, an interleaved video stream, which can be used to determine distance to an object. The video stream may be encoded using a motion estimation encoder. Output of the video encoder may be processed (e.g., parsed) in order to extract motion information present in the encoded video. The motion information may be utilized in order to determine a depth of visual scene, such as by using binocular disparity between two or more images by an adaptive controller in order to detect one or more objects salient to a given task. In one variant, depth information is utilized during control and operation of mobile robotic devices.

Apparatus and methods for distance estimation using multiple image sensors

A sensor bar shelf monitor, for example that may be added to an existing shelving system to convert a store to autonomous operation. The sensor bar may contain distance sensor to detect shoppers reaching towards items on a shelf, and cameras to determine which items shoppers have taken. It may be installed into shelf supports such as gondola shelving uprights. The sensor bar may be located at the front edge of a shelf, and may monitor the shelf below. Placing the sensor bar along the front edge prevents damage to electronics from spills or shelf cleaning, and prevents heat from the sensor bar electronics from damaging items on the shelf. The sensor bar may have a local sensor bar processor that collects sensor data; images may be analyzed locally or transferred to more powerful store processors. A sensor bar may also have controllable lights and controllable electronic labels.

Sensor bar shelf monitor

A robot for capturing image frames of subjects in response to a request includes a base, a robot head, a camera, a vertical positioning mechanism, and a control system. The base has a transport mechanism for controllably positioning the robot along a lateral surface. The vertical positioning mechanism couples the robot head to the base and adjusts a vertical distance between the robot head and the support surface. The control system controls image capture of the camera, the transport mechanism, and the vertical positioning mechanism.

Robotic camera system

A system that integrates camera images and quantity sensors to determine items taken from, placed on, or moved on a shelf or other area in an autonomous store. The items and actions performed may then be attributed to a shopper near the area. Shelves may be divided into storage zones, such as bins or lanes, and a quantity sensor may measure the item quantity in each zone. Quantity changes indicate that a shopper has taken or placed items in the zone. Distance sensors, such as LIDAR, may be used for shelves that push items towards the front. Strain gauges may be used for bins or hanging rods. Quantity changes may trigger analysis of camera images of the shelf to identify the items taken or replaced. Images from multiple cameras that view a shelf may be projected to a vertical plane at the front of the shelf to simplify analysis.

Camera-based tracking and authorization extension system

System that facilitates rapid onboarding of an autonomous (cashier-less) store by capturing images of items in the store's catalog from different angles, with varying backgrounds and lighting conditions, and that automatically builds a classifier training dataset from these images. The system may have cameras in different positions, lights supporting variable illumination, and monitor screens that generate different background colors. It may have an input device such as a barcode reader, and an operator terminal that prompts operators to place items into the imaging system in the necessary orientations. Once an item is placed in the imaging system, a fully automated process may generate a sequence of background colors, a sequence of lighting conditions, and may capture and process images from all of the cameras to create training images. Training images for an item may be generated in seconds, compared to many minutes per item using manual image capture and processing.

Rapid onboarding system for visual item classification

A robotics camera system includes a camera robot that is in wireless communication with a robot cloud server through a cloud network such as the Internet. A user having a mobile device desires to utilize a particular camera robot. The user mobile device may wirelessly communicate with the camera robot either directly or indirectly, through the cloud network. User interaction with the camera robot may be to initiate a photo session and/or to have the camera robot follow the user through a venue. In some embodiments the user initiates the interaction through a social media platform. In some embodiments the user initiates the interaction by utilizing a designated application that runs upon the user's mobile device. In some embodiments the user initiates the interaction with the use of a two-dimensional machine-readable code element. In some embodiments the user initiates an interaction with a plurality of robots that work collaboratively.

Growing community of inventors

San Diego, CA, United States of America

Marius Buibas