Sony Interactive Entertainment Inc.

Orbviu Inc.

Roblox Corporation

Frederick Umminger

Jeffrey Roger Stafford

Jiandong Shen

Crusoe Xiaodong Mao

Brian Michael Christopher Watson

Richard Lee Marks

Steven Osman

Glenn T Black

Philippe Clavel

Sandeep Kanumuri

Todd Tokubo

Eric James Larsen

Noam Rimon

Xiaoyong Ye

Scott Wardle

Hitesh Chhabra

Jason Lawrence Golubock

Palmer Noel Hogen

Plamen Dragozov

Pavel Pavlov

Xia Lou

Joshua Ray Taylor

Paul Kerr

Alex Zhukov

David O'neal

Slawomir Strumecki

Implementations described herein relate to methods, systems, and computer-readable media to provide spatialized audio in virtual experiences. The spatialized audio may be used in voice communications such as, for example, voice and/or video chats. The chats may include spatialized audio that is combined at a client device, or at an online experience platform, and is targeted to a particular user. Individual audio streams may be collected from a plurality of avatars and other objects, and combined based on the target user. The audio may also include background and/or ambient sounds to provide a rich, immersive audio stream in virtual experiences.

Spatialized audio chat in a virtual metaverse

Some implementations relate to methods, systems, and computer-readable media for providing audio for virtual experiences. In some implementations, a method includes receiving, at a server, a first request to generate a plurality of sounds for a user device, wherein the user device is associated with a virtual experience hosted by the server, obtaining, by the server, sound source data for a plurality of sound sources associated with the plurality of sounds, obtaining, by the server, virtual experience state information that comprises a location of a virtual microphone in the virtual experience and at least one of a velocity of the virtual microphone in the virtual experience or an orientation of the virtual microphone in the virtual experience, generating, by the server, an audio mix of the plurality of sounds based on the sound source data and the virtual experience state information, and transmitting the audio mix to the user device.

Distributed processing of sounds in virtual environments

Methods and systems for filtering content include identifying content for presenting in a rendering space defined by one or more mechanisms of a head mounted display (HMD), wherein the content is interactive, streaming content. A rating score for the content is identified. The content is dynamically adjusted to selectively filter a portion of the content based on a user profile of the user, rating score of the content, and ongoing interactions detected at the content. The adjusted content is formatted for rendering on the HMD. The formatted content is automatically transmitted to the HMD for rendering, in response to the request.

Control of personal space content presented via head mounted display

A method and system for layer based encoding of a 360 degrees video is provided. The method includes receiving, by a server, an input video. The input video includes multiple groups of pictures (GOPs). Each GOP starts from a major anchor frame of the input video and includes frames till next major anchor frame. The method also includes generating a first layer. The first layer includes one encoded frame per GOP. The method further includes generating a first sub-layer. The first sub-layer includes encoded frames of multiple mini-GOPs and reconstructed frames of encoded frames of the first layer. Each mini-GOP includes frames between two major anchor frames. Furthermore, the method includes outputting encoded video including the first layer and the first sub-layer.

Method and system for layer based view optimization encoding of 360-degree video

A method and system for constructing view from multiple video streams is provided. The method includes receiving a view independent stream. The method further includes selecting a first view dependent stream, wherein the view independent stream and the first view dependent stream has at least one different geometry. The method also includes generating end user views corresponding to the view independent stream and the first view dependent stream. Further, the method includes blending the end user views to generate a view for display.

Method and system for constructing view from multiple video streams

A method and system for view optimization of a 360 degrees video is provided. The method includes generating two-dimensional video frame from the 360 degrees video. The macroblocks are generated for the two-dimensional video frame. A foveated region of interest for the two-dimensional video frame is defined based on a given view orientation. DCT (Discrete Cosine Transform) coefficients are generated for the macroblocks. View adaptive DCT domain filtering is then performed on the DCT coefficients using the foveated region of interest. Quantization offset is calculated for the DCT coefficients using the foveated region of interest. The DCT coefficients are quantized using the quantization offset to generate encoded two-dimensional video frame for the view orientation. A new view orientation is then set as the given view orientation and steps of generating, performing, calculating, and quantizing are performed for each view orientation and each video frame to generate view optimized video.

Method and system for view optimization of a 360 degrees video

Methods and system for reconstructing 360-degree video is disclosed. A video sequence Vincluding a plurality of frames associated with spherical content at a first frame rate and a video sequence Vincluding a plurality of frames associated with a predefined viewport at a second frame rate is received by a processor. The first frame rate is lower than the second frame rate. An interpolated video sequence V' of the video sequence Vis generated by creating a plurality of intermediate frames between a set of consecutive frames of the plurality of frames of the sequence Vcorresponding to the second frame rate of the video sequence V. A pixel based blending of each intermediate frame of the plurality of the intermediate frames of sequence V&#x2032; with a corresponding frame of the plurality of frames the sequence Vis performed to generate a fused video sequence Vm for displaying.

Method and system for reconstructing 360-degree video

A method for executing computer instructions for presenting an interactive environment in a head-mounted display (HMD) is described. The method includes identifying content associated with the interactive environment to be presented on the HMD for a user and determining whether an interactive object within the identified content satisfies a threshold for presentation to the user. The method includes augmenting the interactive object with augmentation data. The augmented data acts to change a characteristic of the interactive object. The operation of augmenting the interactive object is performed after determining that the interactive object does not satisfy the threshold for presentation to the user. The augmentation data modifies the interactive object to conform the interactive object to be within the threshold.

Filtering and parental control methods for restricting visual activity on a head mounted display

Inertial sensors within a head mounted display are used to track movement of the head mounted display. The tracked movement of the head mounted display is correlated to an action within a virtual reality scene that is currently displayed to a user wearing the head mounted display. The action within the virtual reality scene is based on a context of the virtual reality scene that is currently displayed. The detected movement of the head mounted display can be combined with other sensor data, such as gaze detection data, to determine the action within the virtual reality scene. In this manner, movements of the user as detected using the inertial sensors within the head mounted display are used as inputs to cause actions within the current context of the virtual reality scene as displayed within the head mounted display to the user.

Methods and systems for user interaction within virtual reality scene using head mounted display

A glove interface object is provided, comprising: an index finger portion; a resistance circuit including an index resistor defined along a length of the index finger portion, and a power source for applying a current across the index resistor; a thumb portion; a probe circuit including a thumb contact defined on the thumb portion, the probe circuit configured to read a voltage, via the thumb contact, at a location along the length of the index resistor at which contact between the thumb contact and the index resistor occurs; a data processing module for processing the voltage read by the probe circuit to generate thumb-index control data that is indicative of the location at which the contact between the thumb contact and the index resistor occurs; a communications module configured to send the thumb-index control data to a computing device for processing to define a setting for a virtual environment.

Glove interface object with thumb-index controller

Methods and systems for adjusting a weight of a peripheral device while presenting content to a head mounted display are disclosed. One method includes processing instructions for rendering a virtual reality scene in the head mounted display. The virtual reality scene includes a virtual object. The method also includes receiving pairing of the peripheral device for use with the head mounted display for rendering the virtual reality scene and determining a movement of the virtual object. The method causes sending of instructions to the peripheral device for execution by a controller of the peripheral device to move a weight in the peripheral device. The weight is moved to provide a correlation to the determined movement of the virtual object.

Peripheral devices having dynamic weight distribution to convey sense of weight in HMD environments

A method includes generating an audio signal, generating a control signal that is configured to control a haptic feedback device that is incorporated into a device for delivering audio based on the audio signal to a user, and embedding the control signal in the audio signal by using a pseudorandom signal to form an encoded audio signal. Another method includes receiving a signal that includes an audio signal having an embedded control signal, recovering the control signal from the received signal by using a pseudorandom signal, using the recovered control signal to control a haptic feedback device that is incorporated into a device for delivering audio, recovering the audio signal from the received signal, and using the recovered audio signal to generate audio in the device for delivering audio. Systems perform similar steps, and non-transitory computer readable storage mediums each store one or more computer programs.

Growing community of inventors

Oakland, CA, United States of America

Frederick Umminger