X Development LLC

Intrinsic Innovation LLC

Other

Dirk Holz

Timothy Robert Kelch

Elizabeth Murphy

Daniel Shaffer

Troy Straszheim

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for calibrating a robotic workcell. One of the methods includes obtaining an initial model of a workcell having a plurality of calibration entities including a plurality of robots and a plurality of sensors configured to observe movements by one or more calibration entities. executing a calibration program that generates movement data representing movements by the plurality of robots. A plurality of different constraint pairs are generated from sensor data, the constraint pairs specifying a relationship between poses of calibration entities that are observed in different coordinate frames each defined by a calibration entity and is represented in the sensor data. One or more optimization processes are performed on the plurality of different constraint pairs to generate a plurality of calibration values.

Optimizing calibration with constraints between different coordinate frames

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for measuring and reporting calibration accuracy of robots and sensors assigned to perform a task in an operating environment. One of the methods includes obtaining sensor data of one or more physical robots performing a process in an operating environment; generating, from the sensor data for a first robot of the one or more physical robots, a motion profile representing how the first robot moves while performing the process; obtaining data representing a plurality of candidate virtual robot components, each having a respective virtual motion profile and is a candidate to be included in a virtual representation of the operating environment; performing a motion profile matching process to determine a first virtual robot component from the plurality of candidate virtual robot components that matches the first robot; and adding the first virtual robot component to the virtual representation.

Workcell modeling using motion profile matching and swept profile matching

In one aspect, there is provided a computer-implemented method that includes receiving a request to generate workcell data representing physical dimensions of a workcell having a physical robot arm, executing a calibration program that causes the physical robot arm to move within the workcell and record locations within the workcell at which the robot arm made contact with an object, generating, from the locations within the workcell at which one or more sensors of the robot arm recorded a resistance above a threshold, a representation of physical boundaries in the workcell, obtaining an initial virtual representation of the workcell, and updating the initial virtual representation of the workcell according to the representation of physical boundaries generated from executing the calibration program.

Robotic workspace introspection via force feedback

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for measuring and reporting calibration accuracy of robots and sensors assigned to perform a task in an operating environment. One of the methods includes receiving a request to perform a calibration process for one or more robots in an operating environment; in response, performing the calibration process including executing a calibration program that generates movement data representing movements by the one or more robots within the operating environment; computing a measure of calibration accuracy from the movement data; receiving an input program to be executed in the operating environment; determining that the measure of calibration accuracy does not satisfy an accuracy tolerance of the input program; and in response, generating a notification representing that the measure of calibration accuracy generated from performing the calibration process does not satisfy the accuracy tolerance of the input program.

Detecting robotic calibration accuracy discrepancies

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium that automatically calibrates robots and sensors assigned to perform a task in an operating environment. One of the methods includes obtaining a representation of a robotic operating environment. A user selection of a plurality of components to be configured to operate in the robotic operating environment is received. A mapping is obtained between pairs of components to be calibrated and one or more respective calibration processes to perform to calibrate each pair of components. From the mapping, one or more calibration processes to be performed on pairs of components based on the user selection of the plurality of components is computed. Calibration instruction data describing how to perform the one or more calibration processes to be performed on the pairs of components of the user selection is determined and presented.

Determining robotic calibration processes

Embodiments are provided that include maintaining a map of a plurality of markers in an environment. The map includes a last detection time of each marker of the plurality of markers. The embodiments also include receiving a set of detected markers from a robotic device that is configured to localize in the environment using the plurality of markers. The embodiments further include updating, in the map, the last detection time of each marker which has a mapped position that corresponds to a detected position of a detected marker in the set of detected markers. The embodiments additionally include identifying, from the plurality of markers in the map, a marker having a last detection time older than a threshold amount of time. The embodiments still further include initiating an action related to the identified marker.

Determining changes in marker setups for robot localization

Examples relate to simultaneous localization and calibration. An example implementation may involve receiving sensor data indicative of markers detected by a sensor on a vehicle located at vehicle poses within an environment, and determining a pose graph representing the vehicle poses and the markers. For instance, the pose graph may include edges associated with a cost function representing a distance measurement between matching marker detections at different vehicle poses. The distance measurement may incorporate the different vehicle poses and a sensor pose on the vehicle. The implementation may further involve determining a sensor pose transform representing the sensor pose on the vehicle that optimizes the cost function associated with the edges in the pose graph, and providing the sensor pose transform. In further examples, motion model parameters of the vehicle may be optimized as part of a graph-based system as well or instead of sensor calibration.

Methods and systems for simultaneous localization and calibration

One method disclosed includes identifying, in a map of markers fixed in an environment, two co-located markers within a threshold distance of each other, where each of the two co-located markers has a non-overlapping visibility region. The method further includes determining a set of detected markers based on sensor data from a robotic device. The method additionally includes identifying, from the set of detected markers, a detected marker proximate to a first marker of the two co-located markers. The method also includes enforcing a visibility constraint based on the non-overlapping visibility region of each of the two co-located markers to determine an association between the detected marker and a second marker of the two co-located markers. The method further includes determining a location of the robotic device in the environment relative to the map based on the determined association.

Robot localization with co-located markers

Embodiments are provided that include receiving sensor data from a sensor positioned at a plurality of positions in an environment. The environment includes a plurality of landmarks. The embodiments also include determining, based on the sensor data, a subset of the plurality of landmarks detected at each of the plurality of positions. The embodiments further include determining, based on the subset of the plurality of landmarks detected at each of the plurality of positions, a detection frequency of each landmark. The embodiments additionally include determining, based on the determined detection frequency of each landmark, a localization viability metric associated with each landmark. The embodiments still further include providing for display, via a user interface, a map of the environment. The map includes an indication of the localization viability metric associated with each landmark.

Landmark placement for localization

Methods and systems for detecting sensor orientation characteristics using marker-based localization are disclosed herein. In one aspect, a robotic device can: receive a map of a horizontal marker plane that includes mapped positions of a first marker and a second marker arranged in the horizontal marker plane; receive, from a sensor configured to scan a two-dimensional sensor plane, sensor data indicative of positions of the first and second markers relative to the sensor; determine measured positions of the first and second markers based on the sensor data and a current position of the sensor; determine a difference vector between a first vector that connects the mapped positions of the first and second markers and a second vector that connects the measured positions of the first and second markers; and determine, based on the difference vector, an orientation of the two-dimensional sensor plane relative to the horizontal marker plane.

Detecting sensor orientation characteristics using marker-based localization

An example method includes determining locations of a plurality of candidate landmarks in relation to a robot based on sensor data from at least one sensor on the robot. The method further includes determining a plurality of sample sets, wherein each sample set comprises a subset of the plurality of candidate landmarks and a plurality of corresponding mapped landmarks. The method also includes determining a transformation for each sample set that relates the candidate landmarks from the subset to the corresponding mapped landmarks. The method additionally includes applying the determined transformation for each sample set to the plurality of candidate landmarks to determine a number of inliers associated with each sample set based on distances between the transformed plurality of candidate landmarks and a plurality of neighbouring mapped landmarks. The method further includes selecting a sample set from the plurality based on the number of inliers associated with each sample set. The method still further includes estimating a pose of the robot based on the selected sample set.

Localization of robotic vehicles

One method disclosed includes determining a map of markers fixed in an environment, where the map of markers includes a location and an orientation of each marker. The method further includes determining locations of a set of detected markers relative to the map based on a location of a robotic device relative to the map and based on sensor data from the robotic device. The method also includes associating a detected marker from the set of detected markers with a mapped marker based on the determined location of the detected marker relative to the map and based on a visibility constraint related to the orientation of the mapped marker. The method additionally includes adjusting, in the map, the orientation of the mapped marker based on the determined location of the detected marker relative to the map.

Determining the orientation of flat reflectors during robot mapping

Example embodiments include determining a map of an environment of a robotic vehicle. The map includes locations of a plurality of mapped landmarks within the environment and a false detection source region within the environment. The embodiments further include detecting a plurality of candidate landmarks, and determining which of the detected candidate landmarks correspond to one of the plurality of mapped landmarks and which correspond to false detections. The embodiments additionally include estimating a pose of the robotic vehicle within the environment. The embodiments further include determining which of the detected candidate landmarks determined to correspond to false detections fall within the false detection source region. The embodiments still further include determining a confidence level of the pose estimate based on which of the detected candidate landmarks determined to correspond to false detections fall within the false detection source region.

Growing community of inventors

Mountain View, CA, United States of America

Dirk Holz