Fanuc Corporation

Fanau Ltd.

Tetsuaki Kato

Ryo Nihei

Soichi Arita

Nobutoshi Torii

Te Tang

Atsushi Watanabe

Kaimeng Wang

Hsien-Chung Lin

Yukinobu Tsuchida

Hiroji Nishi

Tomoyuki Yamamoto

Yasuo Naito

Wenjie Chen

Takeaki Aramaki

Toru Shirahata

Atsuo Nagayama

Osamu Yoshida

Masaru Nakamura

Kiyonori Inaba

Yu Zhao

Masakazu Ichinose

Koichi Okanda

Hiroyuki Kawamura

Takashi Sato

Hajime Suzuki

Ichiro Kanno

Masayuki Hamura

Satoshi Kinoshita

Kaname Matsumoto

Kazuhisa Otsuka

Chi-Keng Tsai

Kokoro Hatanaka

Hideki Sugiyama

Chiara Talignani Landi

Makoto Someya

Changhao Wang

Teruki Kuroshita

Xinghao Zhu

Yukinobu Room 6-205 Tsuchida

Makoto Fanuc Dai3virakaramatsu Someya

Yukinobu Fanuc Manhonharimomi Tsuchida

Masakazu Fanuc Dai3virakaramatsu Ichinose

A robot system includes: a feature point position detection unit configured to detect, at a constant cycle, a position of a feature point of an obstacle that moves or deforms within a motion range of a robot; a movement path calculation unit configured to calculate a movement path of the robot before a motion of the robot; a mapping function derivation unit configured to derive a mapping function based on a position of the feature point that is detected at a time interval; and a path adjustment unit configured to dynamically adjust the movement path of the robot using the derived mapping function.

Robot system

A method and system for dynamic collision avoidance motion planning for industrial robots. An obstacle avoidance motion optimization routine receives a planned path and obstacle detection data as inputs, and computes a commanded robot path which avoids any detected obstacles. Robot joint motions to follow the tool center point path are used by a robot controller to command robot motion. The planning and optimization calculations are performed in a feedback loop which is decoupled from the controller feedback loop which computes robot commands based on actual robot position. The two feedback loops perform planning, command and control calculations in real time, including responding to dynamic obstacles which may be present in the robot workspace. The optimization calculations include a safety function which efficiently incorporates both relative position and relative velocity of the obstacles with respect to the robot.

Dynamic motion planning system

A system and method for identifying an object to be picked up by a robot. The method includes obtaining a 2D red-green-blue (RGB) color image and a 2D depth map image of the objects using a 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the objects. The method generates a segmentation image of the objects using a deep learning convolutional neural network that performs an image segmentation process that extracts features from the RGB image, assigns a label to the pixels so that objects in the segmentation image have the same label and rotates the object using the orientation of the object in the segmented image. The method then identifies a location for picking up the object using the segmentation image and the depth map image and rotates the object when it is picked up.

Object bin picking with rotation compensation

A robot collision avoidance motion planning technique using a worst state search and optimization. The motion planning technique begins with a geometric definition of obstacles, start and goal points, and an initial set of waypoints which may be sparsely spaced. Given an inter-point interpolation method such as linear or spline, a continuous trajectory can be described as a function of the waypoints and an arc length parameter. A worst state search is then performed which finds a location between each adjacent pair of waypoints having a worst state of distance to obstacle, considering all parts of the robot and tool. A collision avoidance constraint is defined using the worst state locations, and an optimization of the waypoint locations is then performed to improve the worst states until all collisions are eliminated and an obstacle avoidance minimum distance criteria is met.

Collision avoidance motion planning method for industrial robot

A robot motion planning technique for component assembly operations. Inputs to the motion planning technique include geometric models of the components being assembled, and initial and target configurations. The method begins at a tightly-constrained target or final configuration and plans in the direction of a loosely-constrained initial configuration. A randomly-sampled waypoint configuration is proposed, followed by a local search for feasible configurations which generates nodes that extend a path toward the initial configuration while sliding through the tightly-constrained region. The local search can be repeated multiple times for a given randomly-sampled configuration. When a completed path is found, the action sequence is trimmed to eliminate unnecessary extraneous motions in the loosely-constrained region. The disclosed method dramatically reduces the number of unproductive configurations evaluated and finds assembly solutions much faster in comparison to known tree-based motion planning methods.

Disassembly based assembly planning

A method for teaching a robot to perform an operation based on human demonstration with images from a camera. The method includes a teaching phase where a 2D or 3D camera detects a human hand grasping and moving a workpiece, and images of the hand and workpiece are analyzed to determine a robot gripper pose and positions which equate to the pose and positions of the hand and corresponding pose and positions of the workpiece. Robot programming commands are then generated from the computed gripper pose and position relative to the workpiece pose and position. In a replay phase, the camera identifies workpiece pose and position, and the programming commands cause the robot to move the gripper to pick, move and place the workpiece as demonstrated. A teleoperation mode is also disclosed, where camera images of a human hand are used to control movement of the robot in real time.

Robot teaching by human demonstration

A robot collision avoidance motion optimization technique using a distance field constraint function. CAD or sensor data depicting obstacles in a robot workspace are converted to voxels, and a three-dimensional binary matrix of voxel occupancy is created. A corresponding distance map matrix is then computed, where each cell in the distance map matrix contains a distance to a nearest occupied cell. The distance map matrix is used as a constraint function in a motion planning optimization problem, where the optimization problem is convexified and then iteratively solved to yield a robot motion profile which avoids the obstacles and minimizes an objective function such as distance traveled. The distance field optimization technique is quickly computed and has a computation time which is independent of the number of obstacles. The disclosed optimization technique is easy to set up, as it requires no creation of geometry primitives to approximate robot and obstacle shapes.

Fast robot motion optimization with distance field

A method for dual hand detection in robot teaching from human demonstration. A camera image of the demonstrator's hands and workpieces is provided to a first neural network which determines the identity of the left and right hand of the human demonstrator from the image, and also provides cropped sub-images of the identified hands. The first neural network is trained using images in which the left and right hands are pre-identified. The cropped sub-images are then provided to a second neural network which detects the pose of both the left and right hand from the images, where the sub-image for the left hand is horizontally flipped before and after the hand pose detection if second neural network is trained with right hand images. The hand pose data is converted to robot gripper pose data and used for teaching a robot to perform an operation through human demonstration.

Dual hand detection in teaching from demonstration

A robot optimization motion planning technique using a refined initial reference path. When a new path is to be computed using motion optimization, a candidate reference path is selected from storage which was previously computed and which has similar start and goal points and collision avoidance environment constraints to the new path. The candidate reference path is adjusted at all state points along its length to account for the difference between the start and goal points of the new path compared to those of the previously-computed path, to create the initial reference path. The initial reference path, adjusted to fit the start and goal points, is then used as a starting state for the motion optimization computation. By using an initial reference path which is similar to the final converged new path, the optimization computation converges more quickly than if a na&#xef;ve initial reference path is used.

Initial reference generation for robot optimization motion planning

An adaptive robot grasp planning technique for bin picking. Workpieces in a bin having random positions and poses are to be grasped by a robot and placed in a goal position and pose. The workpiece shape is analyzed to identify a plurality of robust grasp options, each grasp option having a position and orientation. The workpiece shape is also analyzed to determine a plurality of stable intermediate poses. Each individual workpiece in the bin is evaluated to identity a set of feasible grasps, and the workpiece is moved to the goal pose if such direct movement is possible. If direct movement is not possible, a search problem is formulated, where each stable intermediate pose is a node. The search problem is solved by evaluating the feasibility and optimality of each link between nodes. Feasibility of each link is evaluated in terms of collision avoidance constraints and robot joint motion constraints.

Adaptive grasp planning for bin picking

A method and system for adapting a CNC machine tool path from a nominal workpiece shape to an actual workpiece shape. The method includes defining a grid of feature points on a nominal workpiece shape, where the feature points encompass an area around the machine tool path but do not necessarily include points on the machine tool path. A probe is used to detect locations of the feature points on an actual workpiece. A space mapping function is computed as a transformation from the nominal feature points to the actual feature points, and the function is applied to the nominal tool path to compute a new tool path. The new tool path is used by the CNC machine to operate on the actual workpiece. The feature points are used to characterize the three dimensional shape of the working surface of the actual workpiece, not just a curve or outline.

Adaptive path generation for CNC machining

A system and method for extracting features from a 2D image of an object using a deep learning neural network and a vector field estimation process. The method includes extracting a plurality of possible feature points, generating a mask image that defines pixels in the 2D image where the object is located, and generating a vector field image for each extracted feature point that includes an arrow directed towards the extracted feature point. The method also includes generating a vector intersection image by identifying an intersection point where the arrows for every combination of two pixels in the 2D image intersect. The method assigns a score for each intersection point depending on the distance from each pixel for each combination of two pixels and the intersection point, and generates a point voting image that identifies a feature location from a number of clustered points.

Feature detection by deep learning and vector field estimation

A system and method for obtaining a 3D pose of an object using 2D images from a 2D camera and a learned-based neural network. The neural network extracts a plurality of features on the object from the 2D images and generates a heatmap for each of the extracted features that identify the probability of a location of a feature point on the object by a color representation. The method provides a feature point image that includes the feature points from the heatmaps on the 2D images, and estimates the 3D pose of the object by comparing the feature point image and a 3D virtual CAD model of the object.

3D pose estimation by a 2D camera

A system and method for obtaining a 3D pose of an object using 2D images from multiple 2D cameras. The method includes positioning a first 2D camera so that it is directed towards the object along a first optical axis, obtaining 2D images of the object by the first 2D camera, and extracting feature points from the 2D images from the first 2D camera using a first feature extraction process. The method also includes positioning a second 2D camera so that it is directed towards the object along a second optical axis, obtaining 2D images of the object by the second 2D camera, and extracting feature points from the 2D images from the second 2D camera using a second feature extraction process. The method then estimates the 3D pose of the object using the extracted feature points from both of the first and second feature extraction process.

3D pose detection by multiple 2D cameras

A method for controlling a robot to perform a complex assembly task such as insertion of a component with multiple pins or pegs into a structure with multiple holes. The method uses an impedance controller including multiple reference centers with one set of gain factors. Only translational gain factors are used&#x2014;one for a spring force and one for a damping force&#x2014;and no rotational gains. The method computes spring-damping forces from reference center positions and velocities using the gain values, and measures contact force and torque with a sensor coupled between the robot arm and the component being manipulated. The computed spring-damping forces are then summed with the measured contact force and torque, to provide a resultant force and torque at the center of gravity of the component. A new component pose is then computed based on the resultant force and torque using impedance controller calculations.

Growing community of inventors

Hadano, Japan

Tetsuaki Kato