Microsoft Technology Licensing, LLC

Skype Limited

Gn Resound A/s

David Yuheng Zhao

Pontus Carlsson

Sergei Nikiforov

Andrei Jefremov

Magnus Hemmendorff

Mattias Nilsson

Sergey Sablin

Sergey Silkin

Konrad Hofbauer

Ming-Chieh Lee

Henrik Valdemar Turbell

Ross Garrett Cutler

Martin Ellis

Thang Duong

Joyce S Fang

Zhuangfei Wu

Amy Lu

Matthieu Michel Jean-Alexis Moinard

Victor Miguel Barbero

Willem Bastiaan Kleijn

John C Tang

Alexander Ypma

Gina Danielle Venolia

Renat Vafin

Sriram Srinivasan

Mu Han

Kori M Inkpen

Karlheinz Wurm

Matthew Jason Pope

Sören Vang Andersen

Chakkaradeep Chinnakonda Chandran

Jesus De Vicente Peña

Iouri Simernitski

Ian Anderson

Ermin Kozica

Bert Devries

Wei Chen

Christopher Asgaard Rödbro

Doug Bunting

Bert De Vries

Apparatus and methods are disclosed for using machine learning models with private and public domains. Operations can be applied to transform input to a machine learning model in a private domain that is kept secret or otherwise made unavailable to third parties. In one example of the disclosed technology, a method includes applying a private transform to produce transformed input, providing the transformed input to a machine learning model that was trained using a training set modified by the private transform, and generating inferences with the machine learning model using the transformed input. Examples of suitable transforms that can be employed include matrix multiplication, time or spatial domain to frequency domains, and partitioning a neural network model such that an input and at least one hidden layer form part of the private domain, while the remaining layers form part of the public domain.

Protecting deep learned models

An agent interfaces with a sending computing device and a receiving computing device to automatically adjust one-way or two-way real-time audio and real-time video transmission parameters responsive to changing network conditions and/or application requirements. The agent incorporates a reinforcement learning model that adjusts transmission parameters to maximize an expected value of a sum of future rewards; the expected value of the sum of future rewards is based on a current state of the sending computing, a current action (e.g. a current set of transmission parameters) at the sending computing device and a reward provided by the receiving computing device. The reward is representative of a user-perceived quality of experience at the receiving computing device.

Reinforcement learning in real-time communications

Techniques are described for efficiently embedding frame masks in a video stream. In some solutions, a computer implemented method includes operations for encoding a frame of video data comprising an array of pixels to generate an encoded video frame and transmitting the encoded video frame. The array of pixels can include foreground pixels and background pixels. The foreground pixels can have respective original luma values which are bounded within a first luma range. In certain examples, encoding the frame of video data can include converting the original luma values of the foreground pixels to updated luma values which are bounded within a second luma range. The second luma range can be shifted and/or compressed from the first luma range.

Embedding frame masks in a video stream

A computer implemented method can decode a frame of video data comprising an array of pixels to obtain decoded luma values and decoded chroma values corresponding to the array of pixels, and extract a frame mask based on the decoded luma values. The frame mask can include an array of mask values respectively corresponding to the array of pixels. A mask value indicates whether a corresponding pixel is in foreground or background of the frame. The method can perform a morphological operation to the frame mask to change one or more mask values to indicate their corresponding pixels are removed from the foreground and added to the background of the frame. The method can also identify foreground pixels after performing the morphological operation to the frame mask, and render a foreground image for display based on the decoded luma values and decoded chroma values of the foreground pixels.

Using morphological operations to process frame masks in video content

Multiple data streams are transmitted from a transmitting device via an end-to-end or process-to-process channel to a receiving device. At the transmitting device a media data stream is generated. A maximum packet size for another data stream is determined based on a determined end-to-end or process-to-process bandwidth and using playout information pertaining to the media data, which conveys a desired playout rate of the media data. At the transmitting device, based on the determined maximum packet size, a plurality of data packets of the other data stream is generated, each having a packet size no more than the determined maximum. The media and other data streams are multiplexed onto the channel at the transmitting device, thereby transmitting both of the data streams from the transmitting device to the receiving device via the same channel.

Multiplexing data

According to a disclosed example, a first video stream is captured via a first camera associated with a first communication device engaged in a multi-party video conference. The first video stream includes a plurality of two-dimensional image frames. A subset of pixels corresponding to a first human subject is identified within each image frame of the first video stream. A second video stream is captured via a second camera associated with a second communication device engaged in the multi-party video conference. A composite video stream formed by at least a portion of the second video stream and the subset of pixels of the first video stream is rendered, and the composite video stream is output for display at one or more of the first and/or second communication devices. The composite video stream may provide the appearance of remotely located participants being physically present within the same visual scene.

Video conferencing supporting a composite video stream

A transmitting device for generating a plurality of encoded portions of a video to be transmitted to a receiving device over a network configured to: receive an error message over a feedback channel from the receiving device indicating at least one of said plurality of encoded portions that has been lost at the receiving device; encode a recovery portion responsive to said receiving said error message; and transmit said recovery portion to the receiving device over said network; wherein said error message includes information pertaining to a decoded portion successfully decoded at the receiving device and said recovery portion is encoded relative to said decoded portion.

Video coding

Method, device and computer program product for stabilizing a video signal. A plurality of frames of the video signal are captured using a camera. A motion sensor associated with the camera is used to generate a plurality of samples representing motion of the camera. The samples are used to determine a displacement of the camera between a first time and a second time, wherein the first time corresponds to an exposure time midpoint of a first frame of the video signal and the second time corresponds to an exposure time midpoint of a second frame of the video signal. The determined displacement is used to compensate for motion in the video signal between the first and second frames caused by the motion of the camera, to thereby stabilize the video signal.

Video stabilization

A receiver receives a first data stream and a second data stream from a transmitting device over a packet-based communication network, the first data stream being of a first media type (e.g. audio) and the second data stream being of a second media type (e.g. video). The first jitter buffer will buffer each of a plurality of portions of the first data stream as they are received via the receiver, and apply a de-jittering delay before outputting each portion of the first data stream (e.g. audio stream) to be played out through the receiving device. The jitter buffer controller receives information on the second data stream (e.g. video stream), and adapts the de-jittering delay of the first jitter buffer (e.g. audio jitter buffer) in dependence on the information on the second data stream (e.g. information on the video stream).

Adapting a jitter buffer

Method, device and computer program product for estimating quality of a video signal, wherein frames of the video signal are received at receiving means of the device, each frame comprising a plurality of blocks of pixels. A section of a first frame of the video signal, which is correctly received, is used to provide a concealed block for use in place of a block of a second frame of the video signal. A difference measure is determined indicative of a difference between at least some of the pixels surrounding the block in the second frame and the corresponding pixels surrounding the section in the first frame. The difference measure is used to estimate the quality of the second frame of the video signal.

Estimating quality of a video signal

Method, device and computer program product for stabilizing a received video signal. A plurality of frames of the video signal is captured using a camera. The video signal is transmitted as an encoded bit stream. Displacement of the camera between successive frames is determined from a motion indication in the bit stream. The determined displacement is used to compensate for motion in the video signal between successive frames caused by the motion of the camera, to thereby stabilize the video signal.

Received video stabilization

A web application intended to be called by a cloud service is debugged locally in a developer's machine. A debugging session is configured to initiate execution of the web application in its intended cloud context by utilizing a reverse proxy mechanism to forward network traffic from a calling cloud service to the developer's machine. In a first embodiment, an integrated development environment (IDE) is able to establish a connection with the reverse proxy mechanism during the debugging session. In a second embodiment, a separate process is used to establish the connection with the reverse proxy mechanism where the separate process is outside of the IDE.

Debugging a web application locally during development

Growing community of inventors

Enebyberg, Sweden

David Yuheng Zhao