Nvidia Corporation

Microsoft Technology Licensing, LLC

Stanley Thomas Birchfield

Jonathan Tremblay

Thang Hong To

Varun Jampani

Avronil Bhattacharjee

David Jesus Acuna Marrero

Aayush Prakash

Mark A Brophy

Cem Anil

Ross Garrett Cutler

Cha Zhang

Nikolai Smolyanskiy

Charles T Loop

Shalini De Mello

Adam G Kirk

Stephen Walter Tyree

Alexey Kamenev

Oliver Arthur Whyte

Jeffrey David Smith

Brian Kai Peasley

Adarsh Prakash Murthy Kowdle

Duncan J McKay

Shariq Iqbal

Erik Leitch

Jia Cheng

Yunzhi Lin

Jan Kautz

Dieter Fox

Sifei Liu

Harshavardhana Narayana Kikkeri

Mihai R Jalobeanu

Arthur Baker

Balakumar Sundaralingam

Kevin Juho Venalainen

Yuke Zhu

Wonmin Byeon

Hung-Yu Tseng

Kunal Gupta

Bowen Wen

Visak Chadalavada Vijay Kumar

Yiran Zhong

David Hoeller

Yifeng Zhu

Valts Blukis

Taeyeop Lee

Apparatuses, systems, and techniques to generate and select grasp proposals. In at least one embodiment, grasp proposals are generated and selected using one or more neural networks, based on, for example, a latent code corresponding to an object.

Grasp pose prediction

Apparatuses, systems, and techniques are described that solve task and motion planning problems. In at least one embodiment, a task and motion planning problem is modeled using a geometric scene graph that records positions and orientations of objects within a playfield, and a symbolic scene graph that represents states of objects within context of a task to be solved. In at least one embodiment, task planning is performed using symbolic scene graph, and motion planning is performed using a geometric scene graph.

Machine learning model for task and motion planning

In at least one embodiment, under the control of a robotic control system, a gripper on a robot is positioned to grasp a 3-dimensional object. In at least one embodiment, the relative position of the object and the gripper is determined, at least in part, by using a camera mounted on the gripper.

Robotic control system

Apparatuses, systems, and techniques to determine a pose and relative dimensions of an object from an image. In at least one embodiment, a pose and relative dimensions of an object are determined from an image based at least in part on, for example, features of the image.

Single-stage category-level object pose estimation

Apparatuses, systems, and techniques to map coordinates in task space to a set of joint angles of an articulated robot. In at least one embodiment, a neural network is trained to map task-space coordinates to joint space coordinates of a robot by simulating a plurality of robots at various joint angles, and determining the position of their respective manipulators in task space.

Transformation of joint space coordinates using machine learning

A manifold voxel mesh or surface mesh is manufacturable by carving a single block of material and a non-manifold mesh is not manufacturable. Conventional techniques for constructing or extracting a surface mesh from an input point cloud often produce a non-manifold voxel mesh. Similarly, extracting a surface mesh from a voxel mesh that includes non-manifold geometry produces a surface mesh that includes non-manifold geometry. To ensure that the surface mesh includes only manifold geometry, locations of the non-manifold geometry in the voxel mesh are detected and converted into manifold geometry. The result is a manifold voxel mesh from which a manifold surface mesh of the object may be extracted.

Generation of differentiable, manifold meshes of arbitrary genus

Training deep neural networks requires a large amount of labeled training data. Conventionally, labeled training data is generated by gathering real images that are manually labelled which is very time-consuming. Instead of manually labelling a training dataset, domain randomization technique is used generate training data that is automatically labeled. The generated training data may be used to train neural networks for object detection and segmentation (labelling) tasks. In an embodiment, the generated training data includes synthetic input images generated by rendering three-dimensional (3D) objects of interest in a 3D scene. In an embodiment, the generated training data includes synthetic input images generated by rendering 3D objects of interest on a 2D background image. The 3D objects of interest are objects that a neural network is trained to detect and/or label.

Neural network model trained using generated synthetic images

One or more images (e.g., images taken from one or more cameras) may be received, where each of the one or more images may depict a two-dimensional (2D) view of a three-dimensional (3D) scene. Additionally, the one or more images may be utilized to determine a three-dimensional (3D) representation of a scene. This representation may help an entity navigate an environment represented by the 3D scene.

Determining a three-dimensional representation of a scene

When an image is projected from 3D, the viewpoint of objects in the image, relative to the camera, must be determined. Since the image itself will not have sufficient information to determine the viewpoint of the various objects in the image, techniques to estimate the viewpoint must be employed. To date, neural networks have been used to infer such viewpoint estimates on an object category basis, but must first be trained with numerous examples that have been manually created. The present disclosure provides a neural network that is trained to learn, from just a few example images, a unique viewpoint estimation network capable of inferring viewpoint estimations for a new object category.

Few-shot viewpoint estimation

A method, computer readable medium, and system are disclosed for performing autonomous path navigation using deep neural networks. The method includes the steps of receiving image data at a deep neural network (DNN), determining, by the DNN, both an orientation of a vehicle with respect to a path and a lateral position of the vehicle with respect to the path, utilizing the image data, and controlling a location of the vehicle, utilizing the orientation of the vehicle with respect to the path and the lateral position of the vehicle with respect to the path.

Performing autonomous path navigation using deep neural networks

Generation of synthetic images for training a neural network model

An object detection neural network receives an input image including an object and generates belief maps for vertices of a bounding volume that encloses the object. The belief maps are used, along with three-dimensional (3D) coordinates defining the bounding volume, to compute the pose of the object in 3D space during post-processing. When multiple objects are present in the image, the object detection neural network may also generate vector fields for the vertices. A vector field comprises vectors pointing from the vertex to a centroid of the object enclosed by the bounding volume defined by the vertex. The object detection neural network may be trained using images of computer-generated objects rendered in 3D scenes (e.g., photorealistic synthetic data). Automatically labelled training datasets may be easily constructed using the photorealistic synthetic data. The object detection neural network may be trained for object detection using only the photorealistic synthetic data.

Detecting and estimating the pose of an object using a neural network model

Stereo matching generates a disparity map indicating pixels offsets between matched points in a stereo image pair. A neural network may be used to generate disparity maps in real time by matching image features in stereo images using only 2D convolutions. The proposed method is faster than 3D convolution-based methods, with only a slight accuracy loss and higher generalization capability. A 3D efficient cost aggregation volume is generated by combining cost maps for each disparity level. Different disparity levels correspond to different amounts of shift between pixels in the left and right image pair. In general, each disparity level is inversely proportional to a different distance from the viewpoint.

Neural network system for stereo image matching

Growing community of inventors

Sammamish, WA, United States of America

Stanley Thomas Birchfield