Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Meta Platforms, Inc.

Vamsi Krishna Ithapu

William Owen Brimijoin, Ii

Philip Robinson

Henrik Gert Hassager

Sang-Ik Cho

Antonio John Miller

Morteza Khaleghimeybodi

Carl Schissler

Majid Mirbagheri

Jacob Ryan Donley

Vladimir Tourbabin

Pablo Francisco Faundez Hoffmann

Peter Harty Dodds

Sebastiá Vicenç Amengual Gari

Kristen Lorraine Grauman

Sebastia Vicenc Amengual Gari

Senthil Purushwalkam Shiva Prakash

Scott Colburn

Abhinav Gupta

The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing multiple modalities to generate accurate two-dimensional floorplans based on sparse digital videos depicting three-dimensional space. In particular, in one or more embodiments, the disclosed systems extract both visual and audio information from sparse digital video coverage of portions of a three-dimensional space and utilize the extracted visual and audio information to generate a two-dimensional floorplan representing both viewed and unviewed portions of the three-dimensional space. For example, the disclosed systems utilize self-attention layers of a specialized machine learning model to maintain and leverage bi-directional relationships among sequences of visual and audio features to generate floorplan predictions associated with the three-dimensional space. The disclosed systems then combine the predictions to generate the two-dimensional floorplan including a geometric layout and one or more semantic room labels.

Generating digital floorplans from sparse digital video utilizing an audio-visual floorplan reconstruction machine learning model

A wearable device includes an audio system. In one embodiment, the audio system includes a sensor array that includes a plurality of acoustic sensors. When a user wears the wearable device, the audio system determines an acoustic transfer function for the user based upon detected sounds within a local area surrounding the sensor array. Because the acoustic transfer function is based upon the size, shape, and density of the user's body (e.g., the user's head), different acoustic transfer functions will be determined for different users. The determined acoustic transfer functions are compared with stored acoustic transfer functions of known users in order to authenticate the user of the wearable device.

Wearer identification based on personalized acoustic transfer functions

An audio system generates customized head-related transfer functions (HRTFs) for a user. The audio system receives an initial set of estimated HRTFs. The initial set of HRTFs may have been estimated using a trained machine learning and computer vision system and pictures of the user's ears. The audio system generates a set of test locations using the initial set of HRTFs. The audio system presents test sounds at each of the initial set of test locations using the initial set of HRTFs. The audio system monitors user responses to the test sounds. The audio system uses the monitored responses to generate a new set of estimated HRTFs and a new set of test locations. The process repeats until a threshold accuracy is achieved or until a set period of time expires. The audio system presents audio content to the user using the customized HRTFs.

Selecting spatial locations for audio personalization

Embodiments relate to calibrating head-related transfer functions (HRTFs) for a user of an audio system (e.g., as a component of a headset) using cartilage conducted sounds. A test sound is presented to a user using a transducer (e.g., cartilage conduction) and an audio signal is responsively received via a microphone at an entrance to the user's ear canal. The test sound and audio signal combination may be provided to an audio server where a model is used to determine one or more HRTFs for the user. Information describing the one or more HRTFs is provided to the audio system to be used for providing audio to the user. The audio server may also use a model to determine geometric information describing a pinna of the user based on the combination. In one embodiment, the geometric information is used to determine the one or more HRTFs for the user.

Head-related transfer function determination using cartilage conduction

A system for generating individualized HRTFs that are customized to a user of a headset. The system includes a server and an audio system. The server determines the individualized HRTFs based in part on acoustic features data (e.g., image data, anthropometric features, etc.) of the user and a template HRTF. The server provides the individualized HRTFs to the audio system. The audio system presents spatialized audio content to the user using the individualized HRTFs.

Individualization of head related transfer functions for presentation of audio content

A method for generating an individualized audio output response for a headset based on images of a user's ear. An image of a portion of a user's head including at least the user's ear is received, the user in the image wearing a headset including a plurality of visual markers. One or more features describing the user's ear are identified based at least in part on a position of one of the plurality of visual markers relative to the user's ear that are used to disambiguate orientation and/or scale of the features in the image. The one or more features are input to a model, and the model is configured to determine an audio output response of the user based on the extracted one or more features. An individualized audio output response is generated for the user based on the audio output response, the individualized audio output response configured to adjust one or more acoustic parameters of audio content provided to the user by the headset.

Personalized equalization of audio output using visual markers for scale and orientation disambiguation

Individualization of head related transfer function templates for presentation of audio content

A system generates an output audio signal for an object or virtual object using image data of a room to select a room impulse response from a database. A headset may include a depth camera assembly (DCA) and processing circuitry. The DCA generates depth image data of a room. The processing circuitry determines room parameters such as the dimensions of the room based on the depth image data. A room impulse response for the room is determined based on referencing a database of room impulse responses using the room parameters. An output audio signal is generated by convolving a source audio signal of an object with the room impulse response.

Room acoustic matching using sensors on headset

A method for generating an individualized audio output response for a headset based on a representation of a user's ear. One or more images of a portion of a user's head including at least the user's ear are received. A representation of the user's ear is generated based in part on the one or more images. A simulation of sound propagation from an audio source to the user's ear is performed based on the representation. An individualized audio output response is generated for the user based on the simulation, the individualized audio output response configured to adjust one or more acoustic parameters of audio content provided to the user by the headset.

Personalized equalization of audio output using 3D reconstruction of an ear of a user

A method for generating an individualized audio output response for a headset using a machine learning model. One or more images of a portion of a user's head including at least the user's ear are received. One or more features describing the user's ear are identified based on the one or more images. The features are input to a model, and the model is configured to determine an audio output response for the user based on the identified one or more features. The audio output response describes one or more acoustic parameters perceived by the user. An individualized audio output response is generated for the user based on the audio output response, the individualized audio output response configured to adjust one or more acoustic parameters of audio content provided to the user by the headset.

Growing community of inventors

Kirkland, WA, United States of America

Vamsi Krishna Ithapu