Facebook, Inc.

Google Inc.

Meta Platforms, Inc.

Zhengping Zuo

Param Reddappagari

Stephane Taine

Brendan Benjamin Aronoff

Hong Zhang

Yaming He

Mayur Kamat

Li Yin

Janahan Vivekanandan

Shawn C P Mei

Vivek Natarajan

Ujjwal Singh

Duckjin Kang

Fuchun Peng

Jinsong Yu

Guowei Shieh

In one embodiment, a method includes receiving at a head-mounted device a speech input from a user and a visual input captured by cameras of the head-mounted device, wherein the visual input comprises subjects and attributes associated with the subjects, and wherein the speech input comprises a co-reference to one or more of the subjects, resolving entities corresponding to the subjects associated with the co-reference based on the attributes and the co-reference, and presenting a communication content responsive to the speech input and the visual input at the head-mounted device, wherein the communication content comprises information associated with executing results of tasks corresponding to the resolved entities.

Processing multimodal user input for assistant systems

In one embodiment, a method includes receiving a user input based on a plurality of modalities at the client system, wherein at least one of the modalities of the user input is a visual modality, determining one or more subjects and one or more attributes associated with the one or more subjects, respectively, based on the visual modality of the user input, resolving one or more entities corresponding to the one or more subjects based on the determined one or more attributes, and presenting a communication content at the client system responsive to the user input, wherein the communication content comprises information associated with executing results of one or more tasks corresponding to the one or more resolved entities.

Embodiments are described for dynamically adapting video encoding to maintain a nearly stable frame rate based on processor capabilities and bandwidth, for example, by varying a quantization parameter. The quality of the encoded video can be varied to maintain the nearly constant frame rate, which may be measured from the number of encoded video frames being transmitted over a network interface.

Dynamic codec adaptation

In one embodiment, a method includes receiving a sharing request to share content generated during a current dialog session from a client system associated with a first user, identifying one or more content objects associated with the sharing request based on a natural-language understanding module, wherein the one or more content objects were previously generated during the current dialog session, determining one or more second users the sharing request is directed to based on a user profile associated with first user, and sending instructions for accessing one or more of the identified content objects to one or more client systems associated with the one or more second users.

Assisting users with efficient information sharing among social connections

A video encoder may be put in a starve mode during a low latency operation in which the video encoder may be operated in a mode that allows video frames to be encoded without any interdependencies such as motion compensation. At least one encoding parameter of the video encoding is selected such that, for each resulting encoded video frame, the video frame fits in exactly one application layer packet.

Video encoding using starve mode

In one embodiment, a method includes receiving from a client system associated with a first user a user input based on one or more modalities, at least one of which is a visual modality, identifying one or more subjects associated with the user input based on the visual modality based on one or more machine-learning models, determining one or more attributes associated with the one or more subjects respectively based on the one or more machine-learning models, resolving one or more entities corresponding to the one or more subjects based on the determined one or more attributes, executing one or more tasks associated with the one or more resolved entities, and sending instructions for presenting a communication content including information associated with the executed one or more tasks responsive to user input to the client system associated with the first user.

Exemplary embodiments relate to techniques for facilitating a connection between two users in a call by providing an 'answering window' when a user calls. When a caller places a call to a callee, the call invitation may remain open for a window of time during which the caller remains available. The callee may opt to answer the call at any time during the window. During the window, the caller may be placed in a lobby and may share audio and/or video. Accepting the call during the answering window may place the callee in the lobby or may directly convert the interaction into an active call. The callee may request that the window can be altered, or may signal their intention to answer the call at a particular time.

Methods and systems for connecting caller and callee client devices

Dynamic codec adaption

Exemplary embodiments relate to techniques for displaying participants in an interface during a group call, and is particularly well-suited to use with small displays. A full-size call interface may be shrunk down into a chathead that makes good use of limited screen real estate, especially when being used on a mobile device. On the smaller chathead, the dominant user is shown, and a roster of users is shrunk and may exhibit reduced functionality. The chathead may be moved around the screen, and other applications may continue to run in the space not occupied by the chathead.

Interface for a video call

A method may include sending, by a client device, an access request to an authentication server device. The access request may include a request to access an administered resource. The method may include in response to the client device not complying with an administrative policy associated with the administered resource, receiving, from the authentication server device, one or more instructions regarding installation of a client application, receiving, by the client device, a client application in accordance with the instructions, and installing the client application on the client device.

Policy enforcement of client devices

Exemplary embodiments relate to techniques for selecting which users should be shown in an interface during a group call, and for presenting the users on (potentially small) displays. According to some embodiments, a most-relevant speaker is selected for display on each call participants' screen. When deciding which user to display in the primary window of a video call, a dominant or relevant user is selected. A dominant user may be selected based on the audio energy represented by the audio packets for the user's device; alternatively dominant user selection may be implemented using artificial intelligence or machine learning, allowing for better differentiation between speaking and noise. On each user's display that does not belong to the relevant user, the current relevant user is shown. On the current relevant user's display, the previous relevant user is shown.

Methods and systems for displaying relevant participants in a video communication

Exemplary embodiments relate to techniques for connecting two users when a caller places a call but a callee rejects the call or fails to answer in a predetermined period of time. The calling application may terminate the call attempt and request status updates regarding the called party to determine when the callee is available for a follow-up call. The system may gain insight into when a user is available based on the user's presence in a messaging or social networking app, activity in a third-party application unrelated to the call, or the power status of the user's device. When it is determined that the callee is available, a notification may be sent to the caller informing the caller that it is a good time to call back. The techniques may also be used in reverse, informing the callee of when the caller is available for a return call.

Growing community of inventors

Medina, WA, United States of America

Zhengping Zuo