Meta Platforms, Inc.

Abhinav Gupta

Philip Robinson

Carl Schissler

Vamsi Krishna Ithapu

Kristen Lorraine Grauman

Sebastia Vicenc Amengual Gari

Senthil Purushwalkam Shiva Prakash

The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing multiple modalities to generate accurate two-dimensional floorplans based on sparse digital videos depicting three-dimensional space. In particular, in one or more embodiments, the disclosed systems extract both visual and audio information from sparse digital video coverage of portions of a three-dimensional space and utilize the extracted visual and audio information to generate a two-dimensional floorplan representing both viewed and unviewed portions of the three-dimensional space. For example, the disclosed systems utilize self-attention layers of a specialized machine learning model to maintain and leverage bi-directional relationships among sequences of visual and audio features to generate floorplan predictions associated with the three-dimensional space. The disclosed systems then combine the predictions to generate the two-dimensional floorplan including a geometric layout and one or more semantic room labels.

Growing community of inventors

Pittsburgh, PA, United States of America

Abhinav Gupta