Facebook, Inc.

Meta Platforms, Inc.

Eric W Hwang

Jason Francis Harrison

Shahid Razzaq

Rahul Nallamothu

Timo Juhani Ahonen

Alan Liu

Josiah Vincent Vivona

Ted Taejin Kim

David Kaufman

Belmer Perrella Garcia Negrillo

Samuel Rash

Dhruba Borthakur

Zheng Shao

Saurabh Mitra

Samuel Franklin Pepose

Jeffrey Zhang

Anshul Kumar Jain

Abhinav Garlapati

Jose M Gonzalez

Lawrence Alan Corwin

In one embodiment, a method includes an intelligent communication device detecting that a person is visible to a camera of the device, determining a first biometric characteristic of the person discernable by the device, associating the first biometric characteristic with a user identifier unique to the person, determining, while the person is identifiable based on the first biometric characteristic, a second biometric characteristic of the person discernable by the device, and associating the second biometric characteristic with the user identifier. The method also includes the device determining that a detected person has a detected biometric characteristic, determining that the detected person is associated with the user identifier by matching the detected biometric characteristic to the first biometric characteristic or the second biometric characteristic, and applying, while the detected person is identifiable based on the detected biometric characteristic, a user-specific customization associated with the user identifier.

User-specific customization of video conferences using multimodal biometric characterization

A video system processes video frames from a wide angle camera to dewarp the video frames in a manner that preserves conformity of an object of interest. A crop region of a video frame corresponding to the object of interest is determined. An input parameter to a dewarping function is generated based on the detected crop region. The dewarping function is applied to the crop region using the input parameter to generate a dewarped video frame and the dewarped video frame is outputted. The input parameter may be generated in a manner that causes the dewarped video frame to have higher conformity and lower distortion in the region around the object of interest than in a region distant from the object of interest.

Adaptive dewarping of wide angle video frames

The disclosed computer-implemented method may include (i) receiving a video stream for encoding, (ii) determining that the video stream is associated with an application, (iii) analyzing the video stream to label one or more regions of a frame within the video stream with a semantic category, (iv) determining, based at least in part on the application with which the video stream is associated, a prioritization of the semantic category, and (v) allocating encoding resources to one or more portions of the frame that comprise at least a part of the one or more regions of the frame based at least in part on the prioritization of the semantic category. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for application- and content-aware real-time video encoding

In one embodiment, a method includes accessing foreground visual data that comprises a set of coordinate points that correspond to a plurality of surface points of a person in an environment; generating a bounding box for the set of coordinate points, wherein the bounding box comprises every coordinate point in the set of coordinate points; providing instructions to collect background visual data for an area in the environment that is outside of the bounding box; and providing the foreground visual data and the background visual data to an intelligent director associated with the computing device.

Automated cinematic decisions based on descriptive models

Various client devices include displays and one or more image capture devices configured to capture video data. Different users of an online system may authorize client devices to exchange information captured by their respective image capture devices. A user viewing video data captured by another user's client device identifies an object of interest in the video data to the other user's client device. The other user's client device modifies captured video data so a focal point of the captured video data is the object of interest and so the object of interest is magnified in the captured video data. Subsequently, the modified video data is transmitted to the client device of the user viewing the captured video data.

Modifying capture of video data by an image capture device based on identifying an object of interest within capturted video data to the image capture device

Multiple users communicate over a network via client devices that include one or more cameras and a display to enable video messaging. At least one of the client devices modifies regions of video data captured by the client device's camera to more prominently identify the people within the video data. To identify a person, the client device disambiguates between actual people and static objects that may appear like people. The client device uses pose models to identify bounding boxes and applies a motion model to determine if a bounding box may include a person based on an amount of movement within the bounding box. If a threshold amount of movement is detected in a bounding box, the client device obtains a higher resolution portion of the scene including the bounding box and classifies whether the bounding box contains a person based on movement within the higher resolution video.

Identifying regions of interest in captured video data objects by detecting movement within higher resolution frames of the regions

Various client devices include displays and one or more image capture devices configured to capture video data. Different users of an online system may authorize client devices to exchange information captured by their respective image capture devices. Additionally, a sending client device captures and transmits video data to a receiving client, while receiving one or more video presentation settings of the receiving client device. The sending client device applies one or more models to the captured video data and compares output from the models to the video presentation settings of the receiving client device. Based on the comparison, the sending client device provides suggested modifications to one or more video presentation settings to the receiving client device. For example, the sending client device provides a suggestion to reorient a display device of the receiving client device.

Modifying presentation of video data by a receiving client device based on analysis of the video data by another client device capturing the video data

In one embodiment, a method includes receiving audio input during an audio-video communication session. The audio input is generated by a first sound source within an environment and a second sound source within the environment. The method includes receiving video input depicting the first sound source and the second sound source in the environment. The method includes identifying the first sound source and the second sound source using the audio input and the video input. The method includes predicting a first engagement metric for the first sound source and a second engagement metric for the second sound source based on the identifying. The method includes processing the audio input to generate an audio output signal based on a comparison of the first engagement metric and the second engagement metric. The method includes providing the audio output signal to a computing device associated with the audio-video communication session.

Audio selection based on user engagement

A client device receives video data and displays the video data via a display device. An overlay including content other than the video data is also displayed in a specific area of the display device and at least partially occludes the video data displayed within the specific area of the display device. The client device identifies coordinates of regions of interest within frames of the video data. When the client device determines that at least a threshold amount of a region of interest within the video data is displayed within the specific area of the display device, where the overlay is displayed, for at least a threshold amount of time, the client device increases a transparency of the overlay, repositions the overlay, of otherwise modifies the overlay to prevent the overlay from occluding the region of interest.

Modifying display of an overlay on video data based on locations of regions of interest within the video data

In one embodiment, a method includes accessing input data from one or more different input sources. The input sources include: one or more cameras, one or more microphones, and a social graph maintained by a social-networking system. Based on the input data, generating a current descriptive model for a current audio-video communication session that comprises one or more descriptive characteristics about (1) an environment associated with the current audio-video communication session, (2) one or more people within the environment, or (3) one or more contextual elements associated with the current audio-video communication session. The method also includes generating one or more instructions for the current audio-video communication session that are based the one or more descriptive characteristics; and sending the one or more instructions to a computing device associated with the one or more cameras and the one or more microphones.

In one embodiment, a method includes identifying, from a set of coordinate points that correspond to a plurality of surface points of a person in an environment, a coordinate point that corresponds to a facial feature of the person; generating a facial structure for a face of the person, wherein the facial structure: covers a plurality of facial features of the person; and substantially matches a pre-determined facial structure; generating a body skeletal structure for the person, wherein the body skeletal structure substantially matches a predetermined body skeletal structure and substantially aligns with the generated facial structure in at least one dimension of a two-dimensional coordinate plane; and associating the generated body skeletal structure and facial structure with the person in the environment.

In one embodiment, a method includes, at a first time during an audio-video communication session: (1) determining that a first participant is located in an environment; (2) locating a first body region of the first participant; (3) generating a first color histogram of the first body region that represents a first distribution of colors of the first body region. The method also includes, at a second time: (1) determining that a second participant is located in the environment; (2) locating a second body region of the second participant; (3) generating a second color histogram of the second body region that represents a second distribution of one or more colors of the second body region. The method also includes determining that the first and second color histograms are substantially alike, and based on the determination, determining that the first participant is the same as the second participant.

In one embodiment, a method includes receiving audio input data from a microphone array of at least two microphones. The audio input data is generated by a first sound source at a first location and a second sound source at a second location. The method also includes calculating a first engagement metric for the first sound source and a second engagement metric for the second sound source. The first engagement metric approximates an interest level of a receiving user for the first sound source, and the second engagement metric approximates an interest level from the receiving user for the second sound source. The method also includes determining that the first engagement metric is greater than the second engagement metric, and processing the audio input data to generate an audio output signal. The audio output signal may amplify sound generated by the first sound source relative to the second sound source.

Techniques for facilitating and accelerating log data processing by splitting data streams are disclosed herein. The front-end clusters generate large amount of log data in real time and transfer the log data to an aggregating cluster. The aggregating cluster is designed to aggregate incoming log data streams from different front-end servers and clusters. The aggregating cluster further splits the log data into a plurality of data streams so that the data streams are sent to a receiving application in parallel. In one embodiment, the log data are randomly split to ensure the log data are evenly distributed in the split data streams. In another embodiment, the application that receives the split data streams determines how to split the log data.

Growing community of inventors

San Jose, CA, United States of America

Eric W Hwang