Sharp Laboratories of America, Inc

Apple Inc.

Sharp Kabushiki Kaisha Corporation

Hao Pan

Yeping Su

Scott J Daly

Jim Chen Chou

Xiao-Fan Feng

Guy Cote

Haitao Guo

Hsi-Jung Wu

Arthur Yasheng Zhang

Baoxin Li

Tobias Eble

Munehiro Nakazato

Moinul H Khan

Petrus J L Van Beek

Krishnakanth Rapaka

Xiaohua Yang

Can Jin

Ye Cong

Andrew Bai

Avi Bar-Zeev

James O Normile

Sheng Zhang

Sorin Constantin Cismas

Geoffrey Grant Stahl

Gurjeet S Saund

Ray Ling Chang

Timothy Robert Oriol

Nicolas P Bonnier

Justin A Hensley

Ke Zhang

Ling Su

Chang Yuan

Kenneth I Greenebaum

M Ibrahim Sezan

Xinyu Xu

Haiyan He

Nicolas Pierre Marie Frederic Bonnier

Ganesh G Yadav

Felix C Fernandes

Jiandong Shen

Cody J White

Linfeng Guo

Randall R Rauwendaal

Nicolas Peirre Marie Frederic Bonnier

Sachin G Deshpande

Xin Ru Wang

Louis Joseph Kerofsky

Chris Yoochang Chung

Jiefu Zhai

Aleksandr M Movshovich

Xuemei Zhang

Yingjun Bai

Alexandre Da Veiga

Bruno M Sommer

Wei Li

Douglas Scott Price

Paul V Johnson

Sreeraman Anantharaman

Ke Chen

Holly Gerhard

Jeffrey S Norris

Qiang Wang

William W Sprague

Francesco Iacopino

Yasuo Ozawa

Xu Zhao

Vaidyanath Mani

Xiang Fu

Robert Thomas Held

Ruiqi Hu

Gui Xie

Jin Wook Chang

In one implementation, a method includes receiving a warped image representing simulated reality (SR) content (e.g., to be displayed in a display space), the warped image having a plurality of pixels at respective locations uniformly spaced in a grid pattern in a warped space, wherein the plurality of pixels are respectively associated with a plurality of respective pixel values and a plurality of respective scaling factors indicating a plurality of respective resolutions at a plurality of respective locations of the SR content (e.g., in the display space). The method includes processing the warped image in the warped space based on the plurality of respective scaling factors to generate a processed warped image and transmitting the processed warped image.

Dynamic foveated pipeline

A mixed reality system that includes a device and a base station that communicate via a wireless connection The device may include sensors that collect information about the user's environment and about the user. The information collected by the sensors may be transmitted to the base station via the wireless connection. The base station renders frames or slices based at least in part on the sensor information received from the device, encodes the frames or slices, and transmits the compressed frames or slices to the device for decoding and display. The base station may provide more computing power than conventional stand-alone systems, and the wireless connection does not tether the device to the base station as in conventional tethered systems. The system may implement methods and apparatus to maintain a target frame rate through the wireless link and to minimize latency in frame rendering, transmittal, and display.

Video pipeline

A video encoding system in which pixel data is decomposed into frequency bands prior to encoding. The frequency bands for a slice of a frame may be buffered so that complexity statistics may be calculated across the frequency bands prior to encoding. The statistics may then be used by a rate control component in determining quantization parameters for the frequency bands for modulating the rate in the encoder for the current slice. The quantization parameters for the frequency bands may be calculated jointly to optimize the quality of the displayed frames after decoder reconstruction and wavelet synthesis on a receiving device. Information about one or more previously processed frames may be used in combination with the statistics for a current slice in determining the quantization parameters for the current slice.

Rate control for low latency video encoding and transmission

A multi-layer low-pass filter is used to filter a first frame of video data representing at least a portion of an environment of an individual. A first layer of the filter has a first filtering resolution setting for a first subset of the first frame, while a second layer of the filter has a second filtering resolution setting for a second subset. The first subset includes a data element positioned along a direction of a gaze of the individual, and the second subset of the frame surrounds the first subset. A result of the filtering is compressed and transmitted via a network to a video processing engine configured to generate a modified visual representation of the environment.

Gaze direction-based adaptive pre-filtering of video data

In one implementation, a method of generating a bit stream encoding a video stream is performed by a device including one or more processors and non-transitory memory. The method includes decomposing a video stream into a plurality of frequency band video streams. The method includes determining a target bitrate and determining, for each frequency band video stream, a respective frequency band bit rate based on the target bit rate. The method includes encoding each of the plurality of frequency band video streams at its respective frequency band bit rate and transmitting, over a channel, each encoded frequency band video stream.

Multi-band rate control

In one implementation, a method of encoding an image is performed at a device including one or more processors and non-transitory memory. The method includes determining a category of a spatial portion of an image based on a relation between a plurality of thresholds associated with a plurality of quantization scaling parameters and a bit rate of the spatial portion of the image at the plurality of quantization scaling parameters. The method includes quantizing the spatial portion of the image based on the categorization.

Video encoder with quantization control

A method includes obtaining first pass-through image data characterized by a first pose. The method includes obtaining respective pixel characterization vectors for pixels in the first pass-through image data. The method includes identifying a feature of an object within the first pass-through image data in accordance with a determination that pixel characterization vectors for the feature satisfy a feature confidence threshold. The method includes displaying the first pass-through image data and an AR display marker that corresponds to the feature. The method includes obtaining second pass-through image data characterized by a second pose. The method includes transforming the AR display marker to a position associated with the second pose in order to track the feature. The method includes displaying the second pass-through image data and maintaining display of the AR display marker that corresponds to the feature of the object based on the transformation.

Methods and devices for detecting and identifying features in an AR/VR scene

A device comprises memory, a display characterized by a display characteristic, and processors coupled to the memory. The processors execute instructions causing the processors to receive data indicative of the display characteristic, data indicative of ambient lighting, and data indicative of content characteristics for a content item; determine a tone mapping curve for the content item based on the data indicative of content characteristics; determine a first, so-called 'anchor' point along the tone mapping curve; modify a first portion of the tone mapping curve below the anchor point based on the data indicative of ambient lighting; modify a second portion of the tone mapping curve above the anchor point based on the data indicative of the display characteristic; perform tone mapping for the content item based on the modified toned mapping curve to obtain a tone mapped content item; and cause the display to display the tone mapped content item.

Ambient headroom adaptation

In one implementation, a method of encoding an image is performed at a device including one or more processors and non-transitory memory. The method includes receiving a first image comprising a plurality of pixels having a respective plurality of pixel locations and a respective plurality of pixel values. The method includes applying a frequency transform to a first spatial portion of the first image to generate a plurality of first frequency coefficients respectively associated with a plurality of spatial frequencies and applying the frequency transform to a second spatial portion of the first image to generate a plurality of second frequency coefficients respectively associated with the plurality of spatial frequencies. The method includes obtaining a first quantization matrix including, for each of the plurality of spatial frequencies, a respective first quantization parameter, wherein a first quantization parameter associated with a first spatial frequency of the plurality of spatial frequencies has a first ratio with a first quantization parameter associated with a second spatial frequency of the plurality of spatial frequencies, and obtaining a second quantization matrix including, for each of the plurality of spatial frequencies, a respective second quantization parameter, wherein a second quantization parameter associated with the first spatial frequency has a second ratio, different than the first ratio, with a second quantization parameter associated with the second spatial frequency. The method includes quantizing the plurality of first frequency coefficients according to the first quantization matrix to generate a plurality of first quantized frequency coefficients and quantizing the plurality of second frequency coefficients according to the second quantization matrix to generate a plurality of second quantized frequency coefficients. The method includes transmitting the plurality of first quantized frequency coefficients and the plurality of second quantized frequency coefficients.

One exemplary implementation involves performing operations at an electronic device with one or more processors and a computer-readable storage medium. The device establishes a wireless communication link with a host device. The device receives, from the host device, a left eye frame and a right eye frame via a sequence of left eye frame transmissions and right eye frame transmissions. The device switches data transmissions schemes according to wireless commination link quality or eye gaze tracking. Adjusting transmission format based on transmission quality of the wireless communication link allows the devices to take advantage of greater bandwidth when available to save power. An additional transmission format is based on alternately transmitting left eye and right eye frames for very low bandwidth.

Adaptive wireless transmission schemes

Growing community of inventors

Camas, WA, United States of America

Hao Pan