Google Inc.

Stefan Welker

Manuel Christian Clement

Johnny Hsienchow Lee

Charles L Chen

Steven L Goldberg

Thor Lewis

Ian MacGillivray

Andrey Doronichev

Darwin Yamamoto

Tim Gleason

Samantha Raja

Shawn Buessing

Shanee Nishry

Daniel Sternfeld

Kevin Horowitz

Corentin Fatus

An example method for providing a graphical user interface (GUI) of a computing device includes receiving an input indicating a target pose of the robot, providing for display on the GUI of the computing device a transparent representation of the robot as a preview of the target pose in combination with the textured model of the robot indicating the current state of the robot, generating a boundary illustration on the GUI representative of a limit of a range of motion of the robot, based on the target pose extending the robot beyond the boundary illustration, modifying characteristics of the transparent representation of the robot and of the boundary illustration on the GUI to inform of an invalid pose, and based on the target pose being a valid pose, sending instructions to the robot causing the robot to perform the target pose.

Methods and systems for graphical user interfaces to control remotely located robots

Utilization of user interface inputs, from remote client devices, in controlling robot(s) in an environment. Implementations relate to generating training instances based on object manipulation parameters, defined by instances of user interface input(s), and training machine learning model(s) to predict the object manipulation parameter(s). Those implementations can subsequently utilize the trained machine learning model(s) to reduce a quantity of instances that input(s) from remote client device(s) are solicited in performing a given set of robotic manipulations and/or to reduce the extent of input(s) from remote client device(s) in performing a given set of robotic operations. Implementations are additionally or alternatively related to mitigating idle time of robot(s) through the utilization of vision data that captures object(s), to be manipulated by a robot, prior to the object(s) being transported to a robot workspace within which the robot can reach and manipulate the object.

Efficient robot control based on inputs from remote client devices

An example computer-implemented method includes receiving, from one or more vision components in an environment, vision data that captures features of the environment, including object features of an object that is located in the environment, and prior to a robot manipulating the object: (i) determining based on the vision data, at least one first adjustment to a programmed trajectory of movement of the robot operating in the environment to perform a task of transporting the object, and (ii) determining based on the object features of the object, at least one second adjustment to the programmed trajectory of movement of the robot operating in the environment to perform the task, and causing the robot to perform the task, in accordance with the at least one first adjustment and the at least one second adjustment to the programmed trajectory of movement of the robot.

Optimization of motion paths of a robot using vision data

Systems and methods for identifying locations and controlling devices are provided. For example, a user may indicate a location by aiming at the location from multiple positions in a physical space. The user may also identify a controllable device to control by aiming at the device. Example systems and methods include determining a first position within a three-dimensional space, receiving a first directional input, and determining a first ray based on the first position and first directional input. Example systems and methods also include determining a second position within the three-dimensional space, receiving a second directional input, and determining a second ray based on the second position and second directional input. Example systems and methods may also include identifying a location within a three-dimensional space based on the first ray and the second ray.

Position-based location indication and device control

Systems and methods are described that include generating a virtual environment for display in a head-mounted display device. The virtual environment may include at least one three-dimensional virtual object having a plurality of volumetric zones configured to receive virtual contact. The method may also include detecting a plurality of inputs corresponding to a plurality of actions performed in the virtual environment on the at least one three-dimensional virtual object. Each action corresponds to a plurality of positions and orientations associated with at least one tracked input device. The method may include generating, for each action and while detecting the plurality of inputs, a plurality of prediction models and determining based on the plurality of prediction models in which of the plurality of volumetric zones the at least one tracked input device is predicted to virtually collide.

Volumetric virtual reality keyboard methods, user interface, and interactions

Systems and methods are described for generating a virtual environment including at least one three-dimensional virtual object within a user interface provided in a head mounted display device, detecting a first interaction pattern and a second interaction pattern. In response to detecting the second interaction pattern, a modified version of the three-dimensional virtual object at the first virtual feature is generated according to the first interaction pattern and at the second virtual feature according to the second interaction pattern. The modified version of the three-dimensional virtual object is provided in the user interface in the head mounted display device.

Two-handed object manipulations in virtual reality

Computer-implemented systems and methods are described for configuring a plurality of privacy properties for a plurality of virtual objects associated with a first user and a virtual environment being accessed using a device associated with the first user, triggering for display, in the virtual environment, the plurality of virtual objects to the first user accessing the virtual environment, determining whether at least one virtual object is associated with a privacy setting corresponding to the first user. In response to determining that a second user is attempting to access the one virtual object, a visual modification may be applied to the object based on a privacy setting. The method may also include triggering for display, the visual modification of the at least one virtual object, to the second user while continuing to trigger display of the at least one virtual object without the visual modification to the first user.

System level virtual reality privacy settings

Techniques of generating a virtual environment in a virtual reality system involves changing, within a user interface of the second user, an attribute of an avatar representing the first user while maintaining a spatial position of an object with which the first user is interacting. In this way, the second user may see only non-threatening or otherwise pleasant avatars within their user interface while other users may not perceive any change to the virtual environment as displayed in their respective user interfaces.

Avatars in virtual environments

Techniques of displaying a virtual environment in a HMD involve generating a lighting scheme within a virtual environment configured to reveal a real object in a room in the virtual environment in response to a distance between a user in the room and the real object decreasing while the user is immersed in the virtual environment. Such a lighting scheme protects a user from injury resulting from collision with real objects in a room while immersed in a virtual environment.

Displaying and interacting with scanned environment geometry in virtual reality

Methods and apparatus to use predicted actions in VR environments are disclosed. An example method includes predicting a predicted time of a predicted virtual contact of a virtual reality controller with a virtual object, determining, based on at least one parameter of the predicted virtual contact, a characteristic of a virtual output the object would make in response to the virtual contact, and initiating producing the virtual output before the predicted time of the virtual contact of the controller with the virtual object.

Methods and apparatus to use predicted actions in virtual reality environments

In at least one general aspect, a method can include detecting an interaction with a first object in a virtual reality (VR) environment, producing first contextual information based on user interaction with the first object, determining a second object to display in the VR environment based on the first contextual information, and presenting the second object in the VR environment. The interaction of the user may include at least one of a movement, a speech, or a physiological trait of the user.

Methods and apparatus to determine objects to present in virtual reality environments

Systems, devices, methods, computer-readable storage media, and electronic apparatuses for camera setting adjustment based on predicted environmental factors are provided. An example system includes a camera assembly, at least one processor, and memory storing instructions. When executed by the at least one processor, the instructions may cause the system to determine a direction of movement of the camera assembly. The instructions may also cause the system to predict environmental factors based on the direction of movement. Additionally, the instructions may cause the system to determine camera settings based on the environmental factors. The instructions may also capture an image with the camera assembly using the determined camera settings. In some implementations, the captured image is used to track or one or more entities.

Growing community of inventors

Mountain View, CA, United States of America

Stefan Welker