Microsoft Technology Licensing, LLC

Facebook, Inc.

Bakkama Srinath Reddy

Shyam Sadhwani

Matthew Richard Wozniak

Vladislav Alexandrov

David H Mebane

Dibyajyoti Roy

Amr Martini

Victor Cherepanov

Yongjun Wu

Thomas W Holcomb

Yuechuan Li

Aaron Oneal

Innovations in stream configuration operations and playback operations for adaptive streaming of panoramic video are described. The innovations include features of adaptive streaming of panoramic video with composite pictures. For example, a stream configuration tool splits an input picture of panoramic video into multiple sections and creates multiple composite pictures. A composite picture includes one of the sections as well as a low-resolution version of the input picture. A playback tool reconstructs one or more composite pictures. Under normal operation, the playback tool can use the reconstructed section(s) of the composite picture(s) to render high-quality views of the panoramic video. If the view window dramatically changes, however, or if encoded data for a section is lost or corrupted, the playback tool can use the low-resolution version of the input picture to render lower-quality details for views of the panoramic video, without disruption of playback.

Adaptive panoramic video streaming using composite pictures

Approaches to selectively using start code emulation prevention ('SCEP') on encoded data for media content are described herein. For example, a media encoder selectively performs SCEP processing on encoded data for media content, and sets a value of a syntax element that indicates whether or not to perform SCEP processing on the encoded data. The encoder stores the encoded data for output as part of a bitstream, where the syntax element is signaled in association with the bitstream. A media decoder receives the encoded data, determines, from the value of the syntax element, whether or not to perform SCEP processing on the encoded data, and selectively performs SCEP processing on the encoded data. In this way, the computational cost of scanning operations for SCEP processing can be avoided in many scenarios, and bit rate increases due to insertion of SCEP bytes can be limited.

Selective use of start code emulation prevention

A video encoder or video decoder buffers multiple blocks of a reconstructed picture of a video sequence. The video encoder/decoder performs deblock filtering between at least some of the multiple blocks. As part of the deblock filtering, the video encoder/decoder selectively filters at least some sample values in a diagonal line that crosses a block-boundary intersection between two diagonally adjacent blocks. When filtering sample values at the block-boundary intersection between four blocks, the video encoder/decoder can evaluate characteristics of all four blocks and adjust sample values in a line between diagonally adjacent blocks. If there is a large visual difference between sample values at corner positions of two diagonally adjacent blocks, the difference can be smoothed by filtering sample values in a diagonal line. In this way, the quality of motion-compensated prediction using the reconstructed picture is improved in many cases.

Boundary-intersection-based deblock filtering

Innovations in stream configuration operations and playback operations for adaptive streaming of panoramic video are described. The innovations include features of adaptive streaming of panoramic video with overlapping sections. For example, a stream configuration tool splits an input picture of panoramic video into overlapping sections. A playback tool reconstructs one or more of the sections. Because of overlap between adjacent sections, for gradual changes in a view window, the playback tool can, in effect, preemptively fetch a new section as the view window moves out of a current section into the new section. By the time the view window reaches a non-overlapping part of the adjacent, new section, the playback tool has retrieved encoded data for the new section and can reconstruct the new section. This hides network latency and stream switching latency from the viewer, and disruption of playback is avoided.

Adaptive panoramic video streaming using overlapping partitioned sections

Systems and methods for dual-representation streaming frames include (1) determining that a new endpoint is attempting to join a real-time video session (e.g., by receiving a real-time video stream made up of a series of frames) and (2) adding the endpoint to the real-time video session by (i) transmitting, to the endpoint, an initial frame that takes the form of an independent frame and (ii) after transmitting the initial frame, transmitting a series of subsequent frames consisting of only dependent frames, such that the only independent frame transmitted to the endpoint while the endpoint maintains its connection to the real-time video session is the initial frame. Various other methods, systems, and computer-readable media are also disclosed.

Dual-representation streaming frames

Innovations in reconstruction and rendering of panoramic video are described. For example, a view-dependent operation controller of a panoramic video playback system receives an indication of a view direction for an application and, based at least in part on the view direction, identifies a section of a picture of panoramic video in an input projection. The view-dependent operation controller limits operations of a color converter, video decoder, and/or streaming controller to the identified section. In this way, the panoramic video playback system can avoid performing operations to reconstruct sections of the picture of panoramic video that will not be viewed. As another example, a mapper of a panoramic video playback system re-projects at least some sample values in an input flat projection towards a center location for a view direction, producing an output flat projection, which an application can use to generate one or more screen projections.

Re-projecting flat projections of pictures of panoramic video for rendering by application

Innovations in opportunistic frame dropping for variable-frame-rate encoding of digital video are presented. In general, a computing system selectively drops a frame when the cost of encoding the frame (e.g., in terms of use of computational resources and/or power) is expected to outweigh the benefit of encoding the frame (e.g., in terms of better quality). For example, a frame dropping module detects whether there is significant change in a given frame relative to a control frame, which is a previous frame stored in a control frame buffer. If significant change is detected, the frame dropping module stores the given frame in the control frame buffer, thereby replacing the control frame, and passes the given frame to a video encoder. Otherwise, the frame dropping module drops the given frame without replacing the control frame in the control frame buffer and without passing the given frame to the video encoder.

Opportunistic frame dropping for variable-frame-rate encoding

Approaches to selectively using start code emulation prevention ('SCEP') on encoded data for media content are described herein. For example, a media encoder selectively performs SCEP processing on encoded data for media content, and sets a value of a syntax element that indicates whether or not to perform SCEP processing on the encoded data. The encoder stores the encoded data for output as part of a bitstream, where the syntax element is signaled in association with the bitstream. A media decoder receives the encoded data, determines, from the value of the syntax element, whether or not to perform SCEP processing on the encoded data, and selectively performs SCEP processing on the encoded data. In this way, the computational cost of scanning operations for SCEP processing can be avoided in many scenarios, and bit rate increases due to insertion of SCPE bytes can be limited.

Innovations in reconstruction and rendering of panoramic video are described, including the use of a platform rendering engine to provide a screen projection based on a view direction specified for an application through an interface. For example, based at least in part on the view direction specified for the application, at least a section of panoramic video in an input projection is identified. At least some of sample values of the at least a section of the picture of panoramic video in the input projection are mapped to a screen projection. The screen projection is output for display to a buffer for the application. Thus, an application may use panoramic video, including updating a view direction, without itself having to render a screen projection for the panoramic video.

Interface for application-specified playback of panoramic video

View-dependent operations during playback of panoramic video

Aspects extend to methods, systems, and computer program products for video bit stream decoding. Aspects include flexible definition and detection of surface alignment requirements for decoding hardware. Surface alignment requirements can be handled by render cropping (e.g., cropping at a video output device), through adjustment and modification of original syntax values in a video bit stream and relaxed media type negotiation in a software (host) decoder. Resolution changes can be hidden with the aligned surface allocation when applicable. Performance can be improved and power consumption reduced by using hidden resolution changes.

Video bit stream decoding

Aspects extend to methods, systems, and computer program products for transforming video bit streams for parallel decoding. Aspects of the invention can be used to break segment coding structure limitations in video bit streams. Aspects can be used to maximize parallelization of video decoding tasks, including motion compensation processing, to more efficiently utilize multi-core and multi-processor computer systems. Multiple portions of intra-segment data can be processed in parallel to speed up single frame processing. Video communication latency and memory requirements are also reduced.

Growing community of inventors

Redmond, WA, United States of America

Bakkama Srinath Reddy