Mujin, Inc.

Xutao Ye

Rosen Nikolaev Diankov

Russell Islam

Jinze Yu

Jose Jeronimo Moreira Rodrigues

Yoshiki Kanemoto

Huan Liu

Puttichai Lertkultanon

Ziyan Zhou

Hamdi Sahloul

Ahmed Abouelela

Kazuto Murase

Denys Kanunikov

A computing system including a processing circuit in communication with a camera having a field of view. The processing circuit is configured to perform operations related to detecting, identifying, and retrieving objects disposed amongst a plurality of objects. The processing circuit may be configured to perform operations related to object recognition template generation, feature generation, hypothesis generation, hypothesis refinement, and hypothesis validation.

Systems and methods for object detection

Systems and methods for generating an object recognition template

A robot control system and a method for automatic camera calibration is presented. The robot control system includes a control circuit configured to determine all corner locations of an imaginary cube that fits within a camera field of view, and determine a plurality of locations that are distributed on or throughout the imaginary cube. The control circuit is further configured to control a robot arm to move a calibration pattern to the plurality of locations, and to receive a plurality of calibration images corresponding to the plurality of locations, and to determine respective estimates of intrinsic camera parameters based on the plurality of calibration images, and to determine an estimate of a transformation function that describes a relationship between a camera coordinate system and a world coordinate system. The control circuit is further configured to control placement of the robot arm based on the estimate of the transformation function.

Method and system for performing automatic camera calibration for robot control

A computing system including a processing circuit in communication with a robot and a camera having a field of view. The processing circuit obtains image information based on the objects in the field of view and a loading environment, the loading environment which includes loading areas, an object queue, and a buffer zone. The computing system is configured to use the obtained image information in motion planning operations for the retrieval and placement of objects from the object queue into the loading environment. Pallets provided within the loading environment (i.e., within the loading areas) are dedicated to receiving objects having corresponding object type identifiers. The computer system further uses the image information to determine the fill status of pallets existing within the loading environment, and whether new pallets need to be brought into the loading environment and/or swapped out with existing pallets to account for future planning and placement operations.

System and method for robotic object placement

A system and method for operating a robotic system to register unrecognized objects is disclosed. The robotic system may use first image data representative of an unrecognized object to derive an initial minimum viable region (MVR). The robotic system may analyze second image data representative of the unrecognized object to detect a condition representative of an accuracy of the initial MVR. The robotic system may register the initial MVR or an adjustment thereof based on the detected condition.

Robotic system with automated object detection mechanism and methods of operating the same

A control apparatus includes a first information acquiring section that acquires three-dimensional information of a first region of surfaces of a plurality of objects, the information being obtained by imaging or scanning the plurality of objects from a first location; a second information acquiring section that acquires three-dimensional information of a second region of surfaces of the plurality of objects, the information being obtained by imaging or scanning the plurality of objects from a second location; and a combining section that generates information indicating three-dimensional shapes of at least a portion of the surfaces of the plurality of objects, based on the three-dimensional information of the first region acquired by the first information acquiring section and the three-dimensional information of the second region acquired by the second information acquiring section.

Shape information generation apparatus, control apparatus, loading/unloading apparatus, logistics system, non-transitory computer-readable medium, and control method

Systems and methods for processing spatial structure data are provided. The system accesses spatial structure data, which describes object structure, and which has depth information indicative of a plurality of layers for the object structure. The system further extracts, from the spatial structure data, a portion of the spatial structure data representative of one layer of the plurality of layers. The system identifies, from the portion of the spatial structure data, a plurality of vertices that describe a contour of the layer. Additionally, the system identifies convex corners of the layer based on the plurality of vertices and performs object recognition according to the convex corners.

Method and computing system for object identification

A method and computing system for transferring objects between a source repository and a destination repository is provided. The computing system is configured to operate by a combination of pre-planned and image base trajectories to improve speed and reliability of object transfer. The computing system is configured to capture image information of repositories and use the captured information to alter or adjust pre-planned trajectories to improve performance.

Method and computing system for performing robot motion planning and repository detection

A system and method for determining occlusion are presented. The system receives camera data generated by at least one camera, which includes a first camera having a first camera field of view. The camera data is generated when a stack having a plurality of objects is in the first camera field of view, and describes a stack structure formed from at least an object structure for a first object of the plurality of objects. The system identifies a target feature of or disposed on the object structure, and determines a 2D region that is co-planar with and surrounds the target feature. The system determines a 3D region defined by connecting a location of the first camera and the 2D region. The system determines, based on the camera data and the 3D region, a size of an occluding region, and determines a value of an object recognition confidence parameter.

Method and system for determining occlusion within a camera field of view

A robot control system and a method for updating camera calibration is presented. The method comprises the robot control system performing a first camera calibration to determine camera calibration information, and outputting a first movement command based on the camera calibration information for a robot operation. The method further comprises outputting, after the first camera calibration, a second movement command to move a calibration pattern within a camera field of view, receiving one or more calibration images, and adding the one or more calibration images to a captured image set. The method further comprises performing a second camera calibration based on calibration images in the captured image set to determine updated camera calibration information, determining whether a deviation between the camera calibration information and the updated camera calibration information exceeds a defined threshold, and outputting a notification signal if the deviation exceeds the defined threshold.

Method and control system for and updating camera calibration for robot control

A system and method for performing automatic camera calibration is provided. The system receives a calibration image, and determines a plurality of image coordinates for representing respective locations at which a plurality of pattern elements of a calibration pattern appear in a calibration image. The system determines, based on the plurality of image coordinates and defined pattern element coordinates, an estimate for a first lens distortion parameter of a set of lens distortion parameters, wherein the estimate for the first lens distortion parameter is determined while estimating a second lens distortion parameter of the set of lens distortion parameters to be zero, or is determined without estimating the second lens distortion parameter. The system determines, after the estimate of the first lens distortion parameter is determined, an estimate for the second lens distortion parameter based on the estimate for the first lens distortion parameter.

Method and system for performing automatic camera calibration

A system and method for motion planning is presented. The system is configured, when an object is or has been in a camera field of view of a camera, to receive first image information that is generated when the camera has a first camera pose. The system is further configured to determine, based on the first image information, a first estimate of the object structure, and to identify, based on the first estimate of the object structure or based on the first image information, an object corner. The system is further configured to cause an end effector apparatus to move the camera to a second camera pose, and to receive second image information for representing the object's structure. The system is configured to determine a second estimate of the object's structure based on the second image information, and to generate a motion plan based on at least the second estimate.

Method and computing system for performing motion planning based on image information generated by a camera

A system and method for operating a robotic system to scan and register unrecognized objects is disclosed. The robotic system may use an image data representative of an unrecognized object located at a start location to implement operations for transferring the unrecognized object from the start location. While implementing the operations, the robotic system may obtain additional data, including scanning results of one or more portions of the unrecognized object not included in the image data. The robotic system may use the additional data to register the unrecognized object.

Robotic system with automated package scan and registration mechanism and methods of operating the same

A system and method for performing automatic camera calibration is presented. The system communicates with a first camera and a second camera, wherein a transparent platform is disposed between the two cameras. When a 3D calibration pattern is disposed on the platform, the system receives a first set of calibration images from the first camera, and a second set of calibration images from the second camera. The system determines, based on the first set of calibration images, a first set of coordinates for corners of the polyhedron. The system further determines, based on the second set of calibration images, a second set of coordinates for the corners. The system determines, based on the coordinates, a spatial relationship between the first camera and the second camera. The system further uses a description of the spatial relationship to generate a 3D model of an object other than the 3D calibration pattern.

Growing community of inventors

Tokyo, Japan

Xutao Ye