Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Meta Platforms, Inc.

Carl Schissler

Philip Robinson

Andrew Lovitt

Peter Henry Maresh

Sebastià Vicenç Amengual Gari

Sebastiá Vicenç Amengual Gari

Vamsi Krishna Ithapu

William Owen Brimijoin, Ii

Peter Harty Dodds

Kristen Lorraine Grauman

Scott Colburn

Sebastiá Vicenc̨ Amengual Gari

Sebastiá Vicenç Amengual Garí

Abhinav Gupta

Sebastia Vicenc Amengual Gari

Senthil Purushwalkam Shiva Prakash

The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing multiple modalities to generate accurate two-dimensional floorplans based on sparse digital videos depicting three-dimensional space. In particular, in one or more embodiments, the disclosed systems extract both visual and audio information from sparse digital video coverage of portions of a three-dimensional space and utilize the extracted visual and audio information to generate a two-dimensional floorplan representing both viewed and unviewed portions of the three-dimensional space. For example, the disclosed systems utilize self-attention layers of a specialized machine learning model to maintain and leverage bi-directional relationships among sequences of visual and audio features to generate floorplan predictions associated with the three-dimensional space. The disclosed systems then combine the predictions to generate the two-dimensional floorplan including a geometric layout and one or more semantic room labels.

Generating digital floorplans from sparse digital video utilizing an audio-visual floorplan reconstruction machine learning model

Determination of material acoustic parameters for a headset is presented herein. A value of a material acoustic parameter is initialized. A simulation is performed using the value of the material acoustic parameter and a model. The model includes a three-dimensional representation of a local area occupied by the headset. During the simulation, the value of the material acoustic parameter is dynamically modified until a reverberation time calculated based on the modified value of the material acoustic parameter falls within a threshold value of a target reverberation time. The model is updated with the modified value of the material acoustic parameter. The model is used to determine one or more acoustic parameters. Audio content is rendered based on the one or more acoustic parameters so that the audio content appears originating from an object in the local area.

Determination of material acoustic parameters to facilitate presentation of audio content

Determination of a set of acoustic parameters for a headset is presented herein. The set of acoustic parameters can be determined based on a virtual model of physical locations stored at a mapping server. The virtual model describes a plurality of spaces and acoustic properties of those spaces, wherein the location in the virtual model corresponds to a physical location of the headset. A location in the virtual model for the headset is determined based on information describing at least a portion of the local area received from the headset. The set of acoustic parameters associated with the physical location of the headset is determined based in part on the determined location in the virtual model and any acoustic parameters associated with the determined location. The headset presents audio content using the set of acoustic parameters received from the mapping server.

Extrapolation of acoustic parameters from mapping server

Determination of an acoustic filter for incorporating local effects of room modes within a target area is presented herein. A model of the target area is determined based in part on a three-dimensional virtual representation of the target area. In some embodiments, the model is selected from a group of candidate models. Room modes of the target area are determined based on a shape and/or dimensions of the model. The room mode parameters are determined based on at least one of the room modes and the position of a user within the target area. The room mode parameters describe an acoustic filter that as applied to audio content, simulates acoustic distortion at the position of the user and at frequencies associated with the at least one room mode. The acoustic filter is generated at a headset based on the room mode parameter and is used to present audio content.

Determination of an acoustic filter for incorporating local effects of room modes

Determination of acoustic parameters for a headset using a mapping server

A headset generates an output audio signal to provide a virtual sound source for an object or virtual object by using a room impulse response generated by a simulation using a model of a room generated from image data. The headset may include processing circuitry that obtains the model of the room determined based on the image data. The image data includes depth image data from a depth camera assembly and color image data from a color camera. The model includes surfaces of the room and acoustic absorptions of the surfaces. The processing circuitry adjusts audio content presented by the headset based on a room impulse response determined based on one or more simulations of sound propagation between a target position of an object and a position of the headset within the room using the surfaces of the room and the acoustic absorptions of the surfaces.

Room acoustic characterization using sensors

A virtual-reality device displays a virtual scene. The device generates audio data for an area source in the virtual scene. The area source is within a predefined near-field distance from the listener. The device selects sample points from the area source and determines, for each sample, energy contributions to two respective successive shells of spherical shells that extend from the listener to the predefined near-field distance. The two shells enclose the sample point source, and the determined energy contributions correspond to sound originating from the sample point source. For each shell, the device determines a head-related impulse response (HRIR) by combining the determined energy contributions for that shell. The device determines an overall HRIR for the virtual scene by combining the determined HRIRs for the shells and combines the audio data with the overall HRIR. The device transmits the combined audio data to sound-producing elements of the device.

Head-related impulse responses for area sound sources located in the near field

An audio system on a headset presents, to a user, audio content simulating a target artificial reality environment. The system receives audio content from an environment and analyzes the audio content to determine a set of acoustic properties associated with the environment. The audio content may be user generated or ambient sound. After receiving a set of target acoustic properties for a target environment, the system determines a transfer function by comparing the set of acoustic properties and the target environment's acoustic properties. The system adjusts the audio content based on the transfer function and presents the adjusted audio content to the user. The presented adjusted audio content includes one or more of the target acoustic properties for the target environment.

Audio system for artificial reality environment

A system generates an output audio signal for an object or virtual object using image data of a room to select a room impulse response from a database. A headset may include a depth camera assembly (DCA) and processing circuitry. The DCA generates depth image data of a room. The processing circuitry determines room parameters such as the dimensions of the room based on the depth image data. A room impulse response for the room is determined based on referencing a database of room impulse responses using the room parameters. An output audio signal is generated by convolving a source audio signal of an object with the room impulse response.

Room acoustic matching using sensors on headset

A method for scaling audio content using reverberant energy for the placement of virtual audio sources in an artificial reality experience. The method comprises obtaining a set of critical distance parameters. The set of critical distance parameters include at least a reverberation time of a local area and a geometry of the local area. The method further comprises determining a critical distance for the local area based on the set of critical distance parameters and scaling an amplitude of audio content based in part on the critical distance and a distance between a target location in the local area and an artificial reality headset in the local area. The method further comprises presenting the audio content by the headset in accordance with the scaled amplitude.

Scaling of virtual audio content using reverberent energy

An audio system generates virtual acoustic environments with three-dimensional (3-D) sound from legacy video with two-dimensional (2-D) sound. The system relocates sound sources within the video from 2-D to into a 3-D geometry to create an immersive 3-D virtual scene of the video that can be viewed using a headset. Accordingly, an audio processing system obtains a video that includes flat mono or stereo audio being generated by one or more sources in the video. The system isolates the audio from each source by segmenting the individual audio sources. Reverberation is removed from the audio from each source to obtain each source's direct sound component. The direct sound component is then re-spatialized to the 3-D local area of the video to generate the 3-D audio based on acoustic characteristics obtained for the local area in the video.

Growing community of inventors

Redmond, WA, United States of America

Carl Schissler