Microsoft Technology Licensing, LLC

Robert Norman Heitkamp

Philip Andrew Edry

Steven Wilssens

Todd R Manion

Paul J Radek

Ziyad Ibrahim

Steven Marcel Elza Wilssens

Brian C McDowell

John Allen Tardif

Ivan Jelev Tashev

Brian Lloyd Schmidt

Jason S Flaks

Ryan Yonghee Kim

Sunil Kumar Vemula

Ibrahim Ziyad

Alejandro Acero

Michael Baseflug

Wei Guo

Duncan James McKay

Molly Rose Suver

Leo Shing

Hoang T Do

Xudong Ni

Todd Ryun Manion

A system for enabling spatializing audio data is provided. The system analyzes audio data to identify when to generate spatialized audio data. The system can receive incoming audio data including a plurality of channel-based audio signals as well as object-based audio. The system performs an analysis of the audio data and/or metadata associated with the audio data to determine when to generate the spatialized audio data. The system can identify one or more categories associated with the audio data (e.g., stereo, mono, game effect, . . . ) and use the category to determine whether to spatialize the audio data or not spatialize the audio data.

Spatializing audio data based on analysis of incoming audio data

The techniques disclosed herein provide apparatuses and related methods for the communication of spatial audio and related metadata. In some implementations, a source provides prerecorded spatial audio that has embedded metadata. A computing device processes the prerecorded spatial audio to generate an audio codec that is segmented to include a first section of audio data and a second section that includes metadata extracted from the prerecorded spatial audio. The generated audio codec may be received by a device that includes an encoder. The encoder may process the generated audio codec to generate audio data that includes the metadata.

Frame coding for spatial audio data

Methods and devices for correcting warping in spatial audio may include identifying a geometric transform that defines a geometric warping between a first spatial geometric model that represents how sound is produced in a first volumetric space and a second spatial geometric model that represents how sound is produced in a second volumetric space different from the first volumetric space. The methods and devices may include determining an inverse of the geometric transform that compensates for the geometric transform. The methods and devices may include applying the inverse of the geometric transform to a first location in the first spatial geometric model by mapping the first location to a second location in the second spatial geometric model to correct for the geometric warping.

Spatial audio warp compensator

The techniques disclosed herein provide application programming interfaces (APIs) for enabling a system to select a spatialization technology. The APIs also enable a system to balance resources by allocating audio objects to a number of applications executing on a computer system. The system coordinates the audio objects between applications and each application can control the number of objects they individually generate. In some configurations, the system can also fold audio objects across different applications. Different spatialization technologies can be selected based on an analysis of contextual data and policy data. For instance, when a new headphone system is plugged in, the system may switch from Dolby Atmos to the Microsoft HoloLens HRTF spatialization technology. The system can dynamically control a number of generated audio objects and dynamically change a utilized spatialization technology based on changes to a computing environment.

Enhanced adaptive audio rendering techniques

A game engine may generate video and audio content on a per-frame basis. Audio data corresponding to a current frame may be generated to comprise sound-field information independent of a speaker configuration or spatialization technology that may be used to play the associated audio. The sound-field may be generated based on monaural audio data corresponding to a sound produced by an in-game object at the object's position as of the current frame. The sound-field information may be transmitted to a remote computing device for reproduction using a selected, available speaker configuration and spatialization technology.

Game streaming with spatial audio

The present disclosure provides a number of techniques for personalization of spatial audio across a plurality of systems. A participant media system participating in a session may generate two outputs: personalized audio data and depersonalized audio data. When a spectator attempts to spectate the session, a media sharing platform may distribute shared depersonalized audio data to the spectator system. Based on this shared depersonalized audio data, the spectator media system can personalized the audio data to cause an endpoint device to render spectator personalized audio based on HRTF data for the spectator. Accordingly, the spectator personalized audio is personalized and allows a rich, immersive spectating experience that overcome drawbacks associated with receiving personalized audio from participant systems.

Personalization of spatial audio for streaming platforms

A system for enabling a shared three-dimensional ('3D') audio bed available to multiple software applications is provided. The system manages bed metadata defining a number of speaker objects of a 3D audio bed. The bed metadata also associates each speaker object with a location, which in some configurations, is defined by a three-dimensional coordinate system. The bed metadata is communicated to a plurality of applications. The applications can then generate custom 3D audio data that associates individual audio streams with individual speaker objects of the 3D audio bed. The applications can then communicate the custom 3D audio data to a 3D audio bed engine, which causes the processing and rendering of the custom 3D audio data to an output device utilizing a selected spatialization technology. Aspects of the 3D bed can be altered when the spatialization technology or the output device changes.

Shared three-dimensional audio bed

The techniques disclosed herein can enable a system to coordinate the processing of object-based audio and channel-based audio generated by multiple applications. The system determines a spatialization technology to utilize based on contextual data. In some configurations, the contextual data can indicate the capabilities of one or more computing resources. In some configurations, the contextual data can also indicate preferences. The preferences, for example, can indicate user preferences for a type of spatialization technology, e.g., Dolby Atmos, over another type of spatialization technology, e.g., DTSX. Based on the contextual data, the system can select a spatialization technology and a corresponding encoder to process the input signals to generate a spatially encoded stream that appropriately renders the audio of multiple applications to an available output device. The techniques disclosed herein also allow a system to dynamically change the spatialization technologies during use.

Adaptive audio rendering

The techniques disclosed herein provide a high fidelity, rich, and engaging experience for spectators of streaming video services. The techniques disclosed herein enable a system to receive, process and, store session data defining activity of a virtual reality environment. The system can generate recorded video data of the session activity along with rendered spatial audio data, e.g., render the spatial audio in the cloud, for streaming of the video data and rendered spatial audio data to one or more computers. The video data and rendered spatial audio data can provide high fidelity video clips of salient activity of a virtual reality environment. In one illustrative example, the system can automatically create a video from one or more camera positions and audio data that corresponds to the camera positions.

Multiple listener cloud render with enhanced instant replay

Application programing interface for adaptive audio rendering

The subject disclosure is directed towards a technology that may be used in an audio processing environment. Nodes of an audio flow graph are associated with virtual mix buffers. As the flow graph is processed, commands and virtual mix buffer data are provided to audio fixed-function processing blocks. Each virtual mix buffer is mapped to a physical mix buffer, and the associated command is executed with respect to the physical mix buffer. One physical mix buffer mix buffer may be used as an input data buffer for the audio fixed-function processing block, and another physical mix buffer as an output data buffer, for example.

Mix buffers and command queues for audio blocks

The present disclosure provides a number of techniques for personalization of audio for communication to an endpoint device. According to one technique, a cloud-based computing device may receive media data from a media platform, and generate depersonalized audio data based on the media data, the depersonalized audio data including at least one audio component associated with the media platform. The technique may further generate user personalized audio data based on the depersonalized audio data. The user personalized audio data may include at least one audio component personalized based on a unique head-related transfer function (HRTF) data associated with a user or default HRTF data. The user personalized audio may be communicated to the endpoint device for consumption by the user. Such techniques reduce the need for advanced audio processing technologies on client systems and makes personalized audio more available to users.

Remote personalization of audio

The techniques disclosed herein enable a system to coordinate audio objects that are generated by multiple applications. A system can receive contextual data from several applications and dynamically determine an allocation of a number of audio objects for each application based on the contextual data. The allocation can be based on a status of one or more applications, user interactions with one or more applications, and other factors. Policy data can also cause the system to allocate a number of audio objects to one or more applications based on an application type and other factors. For instance, a policy may cause a system to allocate more audio objects to a game application vs. a communications application. As a user interacts with an application, e.g., moves or resizes a user interface, closes an application, increases or decreases a level of interaction, the system can reallocate audio objects to individual applications.

Growing community of inventors

Sammamish, WA, United States of America

Robert Norman Heitkamp