Google Inc.

Waymo LLC

Dillon Richard Cower

Guangyu Zhou

Nirmal Jitendra Patel

Tarek Hefny

Arun Kumar Kandoor

Nicholas Reiter

Brandon Young

Andrew Scherkus

Christian L Duvivier

Riley Adams

Jeff Prouty

Rob Suderman

Aspects of the disclosure provide for generating a visualization of a three-dimensional (3D) world view from the perspective of a camera of a vehicle. For example, images of a scene captured by a camera of the vehicle and 3D content for the scene may be received. A virtual camera model for the camera of the vehicle may be identified. A set of matrices may be generated using the virtual camera model. The set of matrices may be applied to the 3D content to create a 3D world view. The visualization may be generated using the 3D world view as an overlay with the image, and the visualization provides a real-world image from the perspective of the camera of the vehicle with one or more graphical overlays of the 3D content.

Synthesizing three-dimensional visualizations from perspectives of onboard sensors of autonomous vehicles

Implementations described herein relate to methods, systems, and computer-readable media to encode a video. A method includes capturing a video frame that includes a face of a person. The method further includes detecting a face in the video frame. The method further includes segmenting the video frame into a plurality of rectangles, the plurality of rectangles including a face rectangle with pixels corresponding to the face in the video frame. The method further includes packing the video frame based on the plurality of rectangles, wherein a greater number of pixels are allocated to the face rectangle as compared to other rectangles in the plurality of rectangles. The method further includes encoding the video frame with metadata describing the packing.

Face-based frame packing for video calls

A method includes receiving a first facial framework and a first captured image of a face. The first facial framework corresponds to the face at a first frame and includes a first facial mesh of facial information. The method also includes projecting the first captured image onto the first facial framework and determining a facial texture corresponding to the face based on the projected first captured image. The method also includes receiving a second facial framework at a second frame that includes a second facial mesh of facial information and updating the facial texture based on the received second facial framework. The method also includes displaying the updated facial texture as a three-dimensional avatar. The three-dimensional avatar corresponds to a virtual representation of the face.

Puppeteering remote avatar by facial expressions

A method includes receiving a set of video frames that correspond to a video, including a first video frame and a second video frame that each include a face, wherein the second video frame is subsequent to the first video frame. The method further includes performing face tracking on the first video frame to identify a first face resampling keyframe and performing face tracking on the second video frame to identify a second face resampling keyframe. The method further includes deriving an interpolation amount. The method further includes determining a first interpolated face frame based on the first face resampling keyframe and the interpolation amount. The method further includes determining a second interpolated face frame based on the second face resampling keyframe and the interpolation amount. The method further includes rendering an interpolated first face and an interpolated second face. The method further includes displaying a final frame.

Face-based frame rate upsampling for video calls

Implementations described herein relate to methods, systems, and computer-readable media to relight a video. In some implementations, a computer-implemented method includes receiving a plurality of frames of a video. Each video frame includes depth data and color data for a plurality of pixels. The method further includes segmenting each frame based on the depth data to classify each pixel as a foreground pixel or a background pixel. The method further includes setting depth value of each background pixel to a fixed depth value and applying a Gaussian filter to smooth depth value for the plurality of pixels. The method further includes calculating surface normals based on the depth values of the plurality of pixels. The method further includes rendering a relighted frame by adding a virtual light based on the surface normals and the color data.

Video lighting using depth and virtual lights

A method () includes receiving a first facial framework () and a first captured image () of a face (). The first facial framework corresponds to the face at a first frame and includes a first facial mesh () of facial information (). The method also includes projecting the first captured image onto the first facial framework and determining a facial texture () corresponding to the face based on the projected first captured image. The method also includes receiving a second facial framework () at a second frame that includes a second facial mesh () of facial information and updating the facial texture based on the received second facial framework. The method also includes displaying the updated facial texture as a three-dimensional avatar (). The three-dimensional avatar corresponds to a virtual representation of the face.

A method includes receiving a first video stream at a playback device, the first video stream having a first quality, receiving at least two second video streams at the playback device, the at least two second video streams each corresponding to a portion of the first video stream, the at least two second video streams having a second quality, the second quality being a higher quality as compared to the first quality, playing back the first video stream at the playback device, selecting a third video stream from the at least two second video streams based on a view perspective of a user of the playback device, and playing back the third video stream together with the first video stream at the playback device.

Growing community of inventors

Kirkland, WA, United States of America

Dillon Richard Cower