Fanuc Corporation

Tetsuaki Kato

Hsien-Chung Lin

Te Tang

Yongxiang Fan

Yu Zhao

Kaimeng Wang

Chi-Keng Tsai

Chiara Talignani Landi

Yujiao Cheng

Yen Po Liu

A method and system for anomaly detection from object images. A feature extractor provides feature data characterizing part images, including good parts and bad parts. Training data for good parts is used to create a k-nearest neighbors (k-NN) model core set. The feature data includes hundreds of feature vectors, each having hundreds of filter dimensions. A weight value of one is initially assigned to each filter, and test data comprising some good and some bad parts is evaluated by a weighted k-NN module to determine an anomaly score from the weighted feature data. After all test images are evaluated, good and bad data points nearest a threshold are selected and a gradient ascent computation is performed to update the filter weights. Anomaly scoring and gradient ascent are performed iteratively until filter weights are identified which maximize the separation between good and bad scores, thereby eliminating missed detections and false anomalies.

Feature weighting for kNN-based anomaly detection

A system and method for adapting a feature extraction neural network to only identify environment independent features in an image. The method includes modifying weights within a dataset classifier neural network to improve the accuracy of the ability of the dataset classifier neural network to identify that training features images are from training images and test features images are from test images. The method also includes modifying the weights within a feature extraction neural network to reduce the accuracy of the ability of the dataset classifier neural network to identify that the training features images are from the training images and the test features images are from the test images.

Method for neural network adaptation

A method for generating a dataset of robot motion programs for training a path generation neural network. A large language model is used to configure a task environment and generate code which creates robot simulations. The large language model uses a robot task library and an asset geometry database as inputs. Based on the task and asset inputs and a task instruction, the large language model breaks down the task into steps, then generates code describing robot and object motion to complete the task. The generated code produces robot motions for the task, and a corresponding robot motion program is created and executed in simulation. The simulated robot motion programs are used to generate collision-free robot paths via RRT and/or optimization, and collision-free paths are validated for robot reachability and object placement success. Validated motion programs are added to the dataset and used for training the path generation neural network.

Path dataset generation

A method for anomaly detection from object images. A feature extractor provides feature data characterizing part images, including good parts and bad parts. Training data for good parts is used to create a k-nearest neighbors (k-NN) model core set. Adaptation data for some good and some bad parts is evaluated by a k-NN module to determine an anomaly score from the feature data. A good-vs-bad threshold for the anomaly score is initially set to a low value. When an anomaly score over the threshold is detected, a human inspects the corresponding image to determine whether the subject part is good or bad. If the part is good and the anomaly was false, the k-NN model core set is updated to include the feature data for the subject part, and the threshold is adjusted higher. After every adaptation, the k-NN model is applied to a test data set to evaluate performance.

Threshold and model adaptation for the kNN-based anomaly detection method

A sensorless method for machine tool chatter detection. A motor torque signal is analyzed in the time domain to determine whether a bit is currently cutting workpiece material. When not cutting material, an air-cut reference signal is stored for later use. When cutting material, the motor torque signal is converted to the frequency domain and filtered in a multi-step process. After removal of the air-cut reference signal via spectral subtraction, and removal of spindle harmonic components, additional filtering is performed to address aliasing and encoder error effects. The aliasing filtering removes artificial peaks in the frequency response spectrum resulting from interaction between sampling frequency and cutting frequency. The encoder error filtering removes frequency response peaks related to encoder design and interaction with motor speed. After filtering, indicator criteria are evaluated to detect chatter, and corrective action is taken when chatter is detected.

Filtering of motor signals for chatter detection

A method and system for robot skill learning for high precision assembly tasks employing a compliance controller. A reinforcement learning (RL) controller is pre-trained in an offline mode using human demonstration data, where several repetitions of the demonstration are performed while collecting state and action data for each repetition. The demonstration data is used to pre-train a neural network in the RL controller, with no interaction of the RL controller with the compliance controller/robot system. Following pre-training, the RL controller is moved to online production where it is coupled to the compliance controller/robot system in a self-learning mode. During self-learning, the neural network-based RL controller uses action, state and reward data to continue learning correlations between states and effective actions. Co-training is provided as needed during self-learning, where a human operator overrides the RL controller actions to ensure successful assembly operations, which improves the learned performance of the RL controller.

Efficient method for robot skill learning

A region-based robotic grasp generation technique for machine tending or bin picking applications. Part and gripper geometry are provided as inputs, typically from CAD files, along with gripper kinematics. A human user defines one or more target grasp regions on the part, using a graphical user interface displaying the part geometry. The target grasp regions are identified by the user based on the user's knowledge of how the part may be grasped to ensure that the part can be subsequently placed in a proper destination pose. For each of the target grasp regions, an optimization solver is used to compute a plurality of quality grasps with stable surface contact between the part and the gripper, and no part-gripper interference. The computed grasps for each target grasp region are placed in a grasp database which is used by a robot in actual bin picking operations.

Region-based grasp generation

A robot interference checking motion planning technique using point sets. The technique uses CAD models of robot arms and obstacles and converts the CAD models to 3D point sets. The 3D point set coordinates are updated at each time step based on robot and obstacle motion. The 3D points are then converted to 3D grid space indices indicating space occupied by any point on any part. The 3D grid space indices are converted to 1D indices and the 1D indices are stored as sets per object and per time step. Interference checking is performed by computing an intersection of the 1D index sets for a given time step. Swept volumes are created by computing a union of the 1D index sets across multiple time steps. The 1D indices are converted back to 3D coordinates to define the 3D shapes of the swept volumes and the 3D locations of any interferences.

Point set interference check method and system

An automated technique for robot gripper fingertip design. A workpiece design and a bin shape are provided as inputs, along with a parameterized gripper design. The gripper parameters define the lengths of segments of the fingertips, and the bend angle between fingertip segments. A fingertip shape, defined by selecting parameter values, is used in a simulated picking of parts from many different randomly defined piles of workpieces in the bin. Grasps for the simulated bin picking are pre-defined and provided as input. A score is assigned to the particular fingertip shape based on the average number of leftovers from the many simulated bin picking operations. A new fingertip shape is then defined by selecting new values for the parameters, and the simulations are repeated to assign a score for the new fingertip shape. This process is repeated to suitably sample the parameter range, and a best-performing fingertip shape is identified.

Automatic gripper fingertip design to reduce leftover in random bin picking applications

A method for identifying inaccurately depicted boxes in an image, such as miss detected boxes and partially detected boxes. The method obtains a 2D RGB image of the boxes and a 2D depth map image of the boxes using a 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the boxes. The method generates a segmentation image of the boxes using a neural network by performing an image segmentation process that extracts features from the RGB image and segments the boxes by assigning a label to pixels in the RGB image so that each box in the segmentation image has the same label and different boxes in the segmentation image have different labels. The method analyzes the segmentation image to determine if the image segmentation process has failed to accurately segment the boxes in the segmentation image.

Failure detection and failure recovery for AI depalletizing

A method and system for robotic motion planning which perform dynamic velocity attenuation to avoid robot collision with static or dynamic objects. The technique maintains the planned robot tool path even when speed reduction is necessary, by providing feedback of a computed slowdown ratio to a tracking controller so that the path computation is always synchronized with current robot speed. The technique uses both robot-obstacle distance and relative velocity to determine when to apply velocity attenuation, and computes a joint speed limit vector based on a robot-obstacle distance, a maximum obstacle speed, and a computed stopping time as a function of the joint speed. Two different control structure implementations are disclosed, both of which provide feedback of the slowdown ratio to the motion planner as needed for faithful path following. A method of establishing velocity attenuation priority in multi-robot systems is also provided.

Method of robotic system dynamic velocity modification

A method for tuning the force control parameters for a general robotic assembly operation. The method uses numerical optimization to evaluate different combinations of the parameters for a robot force controller in a simulation environment that is built based on a real-world robotic setup. This method performs autonomous tuning for assembly tasks based on closed loop force control simulation, where random samples from a distribution of force control parameter values are evaluated, and the optimization routine iteratively redefines the parameter distribution to find optimal values of the parameters. Each simulated assembly is evaluated using multiple simulations including random part positioning uncertainties. The performance of each simulated assembly is evaluated by the average of the simulation results, thus ensuring that the selected control parameters will perform well in most possible conditions. Once the parameters have been optimized, they are applied to real robots to perform the actual assembly operation.

Autonomous robust assembly planning

A method for dual arm robot teaching from dual hand detection in human demonstration. A camera image of the demonstrator's hands and workpieces is provided to a first neural network which determines the identity of the left and right hand from the image, and also provides cropped sub-images of the identified hands. The cropped sub-images are provided to a second neural network which detects the poses of both the left and right hand from the images. The dual hand pose data for an entire operation is converted to robot gripper pose data and used for teaching two robot arms to perform the operation on the workpieces, where each hand's motion is assigned to one robot arm. Edge detection from camera images may be used to refine robot motions in order to improve part localization for tasks requiring precision, such as inserting a part into an aperture.

Dual arm robot teaching from dual hand human demonstration

A method for teaching and controlling a robot to perform an operation based on human demonstration with images from a camera. The method includes a demonstration phase where a camera detects a human hand grasping and moving a workpiece to define a rough trajectory of the robotic movement of the workpiece. Line features or other geometric features on the workpiece collected during the demonstration phase are used in an image-based visual servoing (IBVS) approach which refines a final placement position of the workpiece, where the IBVS control takes over the workpiece placement during the final approach by the robot. Moving object detection is used for automatically localizing both object and hand position in 2D image space, and then identifying line features on the workpiece by removing line features belonging to the hand using hand keypoint detection.

Robot teaching by demonstration with visual servoing

A method and system for motion planning for robots with a redundant degree of freedom. The technique computes a collision avoidance motion plan for a robot with a redundant degree of freedom, without artificially constraining the extra degree of freedom. The motion planning is formulated as a quadratic programming optimization calculation having a multi-component objective function and a collision avoidance constraint function. The formulation is efficient enough to compute the motion plan in real time at every robot control cycle. The collision avoidance constraint ensures clearance of all parts of the robot from both static and dynamic obstacles. Objective function terms include minimizing path deviation, joint velocity regularization and robot configuration or pose regularization. Weighting factors on the terms of the objective function are changeable for each control cycle calculation based on obstacle proximity conditions at the time.

Growing community of inventors

Fremont, CA, United States of America

Tetsuaki Kato