Nvidia Corporation

Dieter Fox

Arsalan Mousavian

Fabio Tozeto Ramos

Ankur Handa

Christopher Jason Paxton

Yashraj Shyam Narang

Clemens Eppner

Yu-Wei Chao

Animesh Garg

Balakumar Sundaralingam

Wei Yang

Jonathan Tremblay

Miles Macklin

Tucker Ryer Hermans

Nathan Donald Ratliff

Iretiayo Akinola

Adithyavairavan Murali

Yu Xiang

Adam Harper Fishman

Byron Boots

Maya Cakmak

Isabella Huang

Ming-Yu Liu

Stanley Thomas Birchfield

Shagan Sah

Stan Birchfield

Siva Kumar Sastry Hari

Yevgen Chebotar

Karl Van Wyk

Elena Oleynikova

Viktor Makoviichuk

Alexander James Millane

Taeyeop Lee

Philip Ammirato

Carolyn Linjon Chen

Qian Wan

De-An Huang

Jan Issac

Valts Blukis

Ajay Uday Mandlekar

Caelan Reed Garrett

Pratyusha Sharma

Carlos Florensa

Michelle Lee

Michael Danielczuk

Martin Sundermeyer

Alexander Conrad Lambert

Weiyu Liu

Bowen Wen

Yun-Chun Chen

Chris Paxton

Luyang Zhu

Alexander Lambert

Approaches presented herein provide for a framework to integrate human provided feedback in natural language to update a robot planning cost or value. The natural language feedback may be modeled as a cost or value associated with completing a task assigned to the robot. This cost or value may then be added to an initial task cost or value to update one or more actions to be performed by the robot. The framework can be applied to both real work and simulated environments where the robot may receive instructions, in natural language, that either provide a goal, modify an existing goal, or provide constraints to actions to achieve an existing goal.

Interactive cost corrections with natural language feedback

Apparatuses, systems, and techniques to perform collision-free motion generation (e.g., to operate a real-world or virtual robot). In at least one embodiment, at least a portion of the collision-free motion generation is performed in parallel.

Collision-free motion generation

Approaches provide for performance of a complex (e.g., compound) task that may involve multiple discrete tasks not obvious from an instruction to perform the complex task. A set of conditions for an environment can be determined using captured image data, and the instruction analyzed to determine a set of final conditions to exist in the environment after performance of the instruction. These initial and end conditions are used to determine a sequence of discrete tasks to be performed to cause a robot or automated device to perform the instruction. This can involve use of a symbolic or visual planner in at least some embodiments, as well as a search of possible sequences of actions available for the robot or automated device. A robot can be caused to perform the sequence of discrete tasks, and feedback provided such that the sequence of tasks can be modified as appropriate.

Interpreting discrete tasks from complex instructions for robotic systems and applications

Apparatuses, systems, and techniques to generate and select grasp proposals. In at least one embodiment, grasp proposals are generated and selected using one or more neural networks, based on, for example, a latent code corresponding to an object.

Grasp pose prediction

Apparatuses, systems, and techniques to determine orientation of an objects in an image. In at least one embodiment, images are processed using a neural network trained to determine orientation of an object.

Determining object orientation from an image with machine learning

Approaches presented herein provide for predictive control of a robot or automated assembly in performing a specific task. A task to be performed may depend on the location and orientation of the robot performing that task. A predictive control system can determine a state of a physical environment at each of a series of time steps, and can select an appropriate location and orientation at each of those time steps. At individual time steps, an optimization process can determine a sequence of future motions or accelerations to be taken that comply with one or more constraints on that motion. For example, at individual time steps, a respective action in the sequence may be performed, then another motion sequence predicted for a next time step, which can help drive robot motion based upon predicted future motion and allow for quick reactions.

Reactive interactions for robotic applications and other automated systems

Apparatuses, systems, and techniques to generate a predicted outcome of an object resulting from a robotic component applying a force. In at least one embodiment, a predicted outcome of an object resulting from a robotic component applying a force is generated based on, for example, a neural network.

Predicting grasp outcomes

A technique for training a neural network, including generating a plurality of input vectors based on a first plurality of task demonstrations associated with a first robot performing a first task in a simulated environment, wherein each input vector included in the plurality of input vectors specifies a sequence of poses of an end-effector of the first robot, and training the neural network to generate a plurality of output vectors based on the plurality of input vectors. Another technique for generating a task demonstration, including generating a simulated environment that includes a robot and at least one object, causing the robot to at least partially perform a task associated with the at least one object within the simulated environment based on a first output vector generated by a trained neural network, and recording demonstration data of the robot at least partially performing the task within the simulated environment.

Neural networks to generate robotic task demonstrations

A system may simulate human motion for human-robot interactions, such as may involve a handover of an object. Motion capture can be performed for a hand grasping and moving an object to a location and orientation appropriate for a handover, without a need for a robot to be present or an actual handover to occur. This motion data can be used to separately model the hand and the object for use in a handover simulation, where a component such as a physics engine may be used to ensure realistic modeling of the motion or behavior. During a simulation, a robot control model or algorithm can predict an optimal location and orientation to grasp an object, and an optimal path to move to that location and orientation, using a control model or algorithm trained, based at least in part, using the motion models for the hand and object.

Simulating physical interactions for automated systems

A human pilot controls a robotic arm and gripper by simulating a set of desired motions with the human hand. In at least one embodiment, one or more images of the pilot's hand are captured and analyzed to determine a set of hand poses. In at least one embodiment, the set of hand poses is translated to a corresponding set of robotic-gripper poses. In at least one embodiment, a set of motions is determined that perform the set of robotic-gripper poses, and the robot is directed to perform the set of motions.

Vision-based teleoperation of dexterous robotic system

Apparatuses, systems, and techniques for determining whether collisions will occur in potential paths of an object within a scene. In at least one embodiment, one or more neural networks determine whether collisions will occur in potential paths of an object within a scene based at least in part on point cloud data of the object and the scene.

Object rearrangement using learned implicit collision functions

One embodiment of a method for controlling a robot includes receiving sensor data associated with an environment that includes an object; applying a machine learning model to a portion of the sensor data associated with the object and one or more trajectories of motion of the robot to determine one or more path lengths of the one or more trajectories; generating a new trajectory of motion of the robot based on the one or more trajectories and the one or more path lengths; and causing the robot to perform one or more movements based on the new trajectory.

Techniques for robot control using neural implicit value functions

One common robotic task is the rearrangement of physical objects situated in an environment. This typically involves a robot manipulator picking up a target object and placing the target object in some target location, such as a shelf, cabinet or cubby, and requires the skills of picking, placing and generating complex collision-free motions, oftentimes in a cluttered environment. The present disclosure provides collision detection for object rearrangement using a three-dimensional (3D) scene representation.

Collision detection for object rearrangement using a 3D scene representation

A machine-learning control system is trained to perform a task using a simulation. The simulation is governed by parameters that, in various embodiments, are not precisely known. In an embodiment, the parameters are specified with an initial value and expected range. After training on the simulation, the machine-learning control system attempts to perform the task in the real world. In an embodiment, the results of the attempt are compared to the expected results of the simulation, and the parameters that govern the simulation are adjusted so that the simulated result matches the real-world attempt. In an embodiment, the machine-learning control system is retrained on the updated simulation. In an embodiment, as additional real-world attempts are made, the simulation parameters are refined and the control system is retrained until the simulation is accurate and the control system is able to successfully perform the task in the real world.

Simulation of tasks using neural networks

A robotic system is provided for performing rearrangement tasks guided by a natural language instruction. The system can include a number of neural networks used to determine a selected rearrangement of the objects in accordance with the natural language instruction. A target object predictor network processes a point cloud of the scene and the natural language instruction to identify a set of query objects that are to-be-rearranged. A language conditioned prior network processes the point cloud, natural language instruction, and the set of query objects to sample a distribution of rearrangements to generate a number of sets of pose offsets for the set of query objects. A discriminator network then processes the samples to generate scores for the samples. The samples may be refined until a score for at least one of the sample generated by the discriminator network is above a threshold value.

Semantic rearrangement of unknown objects from natural language commands

Growing community of inventors

Seattle, WA, United States of America

Dieter Fox