Zoom Communications, Inc.

Zoom Video Communications, Inc.

Zhaofeng Jia

Qiyong Liu

Yuhui Chen

Qiang Gao

Wei Wang

Cheng-Lun Hu

Yang Liu

Jiachuan Deng

Xinyu Yao

Zhenghang Gu

Mengfan Zhang

Ye Wang

Zexian Wu

Chunyu Zhang

Xiangming Zhu

Qi Yang

Shaoming Ye

Example methods and systems provide machine-learning assisted acoustic echo cancellation (AEC). The AEC can be used, as an example, to improve the audio quality for online audio and video conferences. A system according to this disclosure includes a pre-trained, machine-learning, AI model designed to detect, in real time, a unitary voice signal, or a signal representing the speech of a single speaker as opposed to that of multiple speakers. A digital signal processing (DSP) algorithm can then detect the echo state, for example, whether distortion results primarily from an echo. Based on these characteristics, the system can, alternatively and automatically apply either a default mode of AEC to the audio signal, or apply a more aggressive mode of AEC.

Machine-learning assisted acoustic echo cancelation

Methods, systems, and apparatus, including computer programs encoded on computer storage media relate to a method for acoustic echo cancellation. The system inputs one or more signal representations into an acoustic echo cancellation network comprising one or more network blocks to generate a mask, each network block comprising one or more convolutional blocks, each convolutional block comprising one or more neural networks. The system combines the mask and a near-end audio signal representation to generate an echo-cancelled audio signal representation. The system generates an echo-cancelled audio signal based on the echo-cancelled audio signal representation.

Real-time low-complexity echo cancellation

Online audio and video conference applications can utilize a noise removal module to eliminate unwanted audio from a participant's speech. A noise removal module can rely on differentiating between human speech versus other audio to filter out noise. However, in some conference environments, participant and non-participant human speech can be present. Artificial intelligence models can be trained to detect both noise and non-participant audio, based on a variety of factors. The models can label captured audio and various noise removal modules can filter noise based on the output of the models.

Dynamic noise and speech removal

Systems and methods for providing voice feedback for video conferences are provided. A computer-implemented method includes joining, by a client device, a video conference hosted by a video conference provider, the video conference having a plurality of participants using a plurality of client devices. With voice feedback enabled, the client device may receive a first audio stream from the video conference provider, the first audio stream associated with a first participant of the plurality of participants and a second audio stream from a user of the client device, the second audio stream comprising a voice of the user of the client device. The client device can play the first audio stream on one or more channels of one or more audio devices connected to the client device and play the second audio stream on the one or more channels of the one or more audio devices.

Voice feedback during video conferences

One example method for ad hoc client audio device support for virtual conferences includes transmitting, via a first communication channel by a virtual conference device connected to a virtual conference hosted by a virtual conference provider, a first signal, the first signal including connection information to enable a network connection between a remote client device and the virtual conference device; receiving, by the virtual conference device via a second communication channel, a request to connect from the remote client device; connecting to the remote client device using the second communication channel; receiving, via the connection to the remote client device, an audio stream from the remote client device, the audio stream captured by a microphone of the remote client device; and providing the audio stream to the virtual conference.

Ad hoc client audio device support for virtual conferences

For online audio/video conferencing applications deployed in an open office environment, using shared conference devices, it can be advantageous to define an acoustic fence. A non-participant audio received from outside the acoustic fence can be considered noise and filtered out before transmission of an audio signal to a far end recipient. Three suppression stages are used to filter the non-participant audio. The first suppression stage uses beamformers for suppression. The second suppression stage is mask-based, and the third suppression stage is reference-based. The three suppression stages filter out non-participant audio signals, having a wide range of frequencies.

Acoustic fence

One example method includes presenting, by a client device, a view of a virtual conference hosted by a virtual conference provider, the virtual conference including a plurality of participants, the client device associated with a participant of the plurality of participants, the view including a plurality of groupings of participants within a virtual conference area, each grouping associated with a different meeting or sub-meeting of the virtual conference; assign a location within the virtual conference area to the participant; receiving, at the client device from the conference provider, one or more audio streams associated with one or more audio sources within the plurality of groupings, the one or more audio streams provided by one or more remote client devices; determining a first location within the virtual conference area of a first audio source of the one or more audio sources; generating a plurality of spatialized audio streams based on the first location of the first audio source, the location of the indicator, and a first audio stream associated with the first audio source; and outputting the spatialized audio streams.

Spatial audio in virtual conference mingling

Example methods and systems provide automatic audio equalization for online conferences. A target, ideal, audio frequency response for use in online conferencing audio can be designed and preset for reference, and the equalizer can operate in real time to reach the target frequency response for any input. The equalizer can selectively apply per-band equalization to the audio signal as needed to adjust an energy value for each of multiple frequency bands to produce an output audio signal that that can be, as an example, routed through meeting servers or other infrastructure to other conference participants. The equalization can compensate for conditions local to a speaker that would otherwise adversely affect audio quality.

Growing community of inventors

Saratoga, CA, United States of America

Zhaofeng Jia