Avid Technology Limited

Other

Robert A Gonsalves

Brian C Cooper

Eric C Peters

Joseph H Rice

Michael D Laird

Chia-Sheng Chou

Timothy H Claman

Robert Alan Paoni

Patrick D O'Connor

Curt A Rawley

Stephen J Reber

Paul D Garmon

Garth Allen Dickie

Peter J Fasciano

Katherine H Cornog

Jeffrey D Kurtze

Albert W Kovalick

Randy M Fayan

Raymond D Cacciatore

Joseph S Plazak

Ronald C Wallace

Sam Butler

Straker J Coniglio

Stephane Daigle

Dion C Scoppettuolo

Daniel C Robbins

Objects within an image are assigned a semantic interest that is indicative of their importance to the image as a whole. The objects are detected automatically, and sub-images that span each of the objects are excerpted from the image. An image embedding is determined for each of the sub-images as well as for the whole image by using an image encoder implemented as a trained multi-modal neural network. The degree of similarity between the image embeddings of a sub-image and that of the whole image is used as a measure of the semantic importance of the object portrayed in the sub-image. Objects of high semantic importance comprise the semantic regions of interest of the image. Knowledge of such regions may be used to enhance and improve the efficiency of downstream image-processing tasks, such as image compression, pan and scan, and contrast enhancement.

Finding the semantic region of interest in images

Automatic editing of media compositions is performed using media editing applications equipped with neural-network-based deep-learning models. The automatic editing is adapted to the practices of local users of a media editing application by training the models on a combination of media compositions previously edited by third-party media editors and media compositions edited by local users. Training data input vectors for the model comprise representative portions of a composition's raw media, and corresponding output vectors include values of parameters that define editing functions applied to the raw media to generate an edited media composition. A user interface enabling a user to adjust and monitor machine learning parameters is provided. Adaptive automatic editing may assist in the creation of video and audio compositions, as well as in the generation of musical scores.

Adaptive deep learning for efficient media content creation and manipulation

Efficient processing of chroma-subsampled video is performed using convolutional neural networks (CNNs) in which the luma and chroma channels are processed separately. The luma channel is independently convolved and downsampled and, in parallel, the chroma channels are convolved and then merged with the downsampled luma to generate encoded chroma-subsampled video. Further processing of the encoded video that involves deconvolution and upsampling, splitting into two sets of channels, and further deconvolutions and upsampling is used in CNNs to generate decoded chroma-subsampled video in compression-decompression applications, to remove noise from chroma-subsampled video, or to upsample chroma-subsampled video to RGBvideo. CNNs with separate luma and chroma processing in which the further processing includes additional convolutions and downsampling may be used for object recognition and semantic search in chroma-subsampled video.

Processing of chroma-subsampled video using convolutional neural networks

A collaboration server hosts software for collaborative composition and editing of a media project with project collaborators using different media editing applications each having their own native data format. Project collaborators, such as video editors, sound editors, effects and graphics artists, and producers access a shared project workspace which contains a snapshot of the current state of the media project in a canonical format, as well as source media files, native application metadata, and change notes. Each editing application includes a module enabling it to read the canonical snapshot representation, and also to flatten its native data model representation into the canonical representation for writing to the shared project workspace. A collaboration server hosts the shared project space, and includes a workflow manager for issuing change notifications and handling versions, and an application server for the shared project user interface. Change notes are generated manually and also expressed automatically in terms of machine-readable change primitives that serve to direct an editor's attention to portions of the media project needing attention.

Methods and systems for collaborative media creation

A computer-based method for media composition of a family of related time-based media programs. The method involves creating a master program with time-based elements of video and/or audio as well as time-based and non-time-based metadata, creating a derivative program that includes derivative elements, defining an inheritance relationship between the master program and the derivative program that specifies elements of the master program to be inherited by the derivative program, and causing the derivative program to inherit the specified elements from the master program in accordance with the inheritance relationship. User interfaces are provided for creating, editing, and viewing hierarchical trees of related programs.

Hierarchical multimedia program composition

A system and method for generating a representation of a color modification to be applied to segments on a digital nonlinear editing system, where each segment is a component of a media composition, and represents a section of a digital media. An indication of a modification to be applied to a color attribute of a segment is received, and the source from which the segment originates is identified. The indication of the color modification is then stored, and, as a result, the color modification is applied to other segments that originate from the identified source. Source color modification is applied to a section of a digital media on a digital nonlinear editing system. a media segment represents the section of a digital media. The segment is a component of a media composition, and originates from a source data structure. The source data structure also represents the section of the digital media. The section is received, and the first source data structure from which the segment originates is identified. It is determined whether the first source data structure includes a color modification attribute, where the color modification attribute defines a color modification to be applied to sections of the digital media represented by segments that originate from the source data structure. If the source data structure includes the color modification attribute, the color modification is applied to the section of media.

Source color modification on a digital nonlinear editing system

Automatic color correction is applied to a scene or clip, including a sequence of images, in a motion picture by selecting a representative image of the scene, analyzing the image and adjusting parameters of a color correction operation that is performed on the sequence of images included in the scene. This operation can be repeated automatically for all scenes or for selected scenes in the motion picture. The parameters may be adjusted to automatically color balance the image while maintaining substantially constant contrast. Analysis of the representative image may include identifying an offset of a peak in a two-dimensional histogram of the colors in the representative image from a white point. Parameters of a color correction operation are adjusted according to this offset. Separate histograms and offsets may be determined for shadows, midtones and highlight regions of the representative image. Analysis of the representative image may include determining a one-dimensional histogram of the luminance information in the representative image. The darkest level and the brightest level in the image are used to balance the image. In particular, the histograms for color channels in the image, such as red, green and blue, are adjusted to match the darkest level and brightest level identified by the luminance histogram.

Automatic color correction for sequences of images

To make an intuitive interface for curves for color correction, a particular color is selected, such as gray, and the effect of each curve on pixels of that color may be displayed as a background image for the curve. A background image also may indicate a color to which tones corresponding to a selected and manipulated point of a curve will be modified or corrected. An image representing a gradient from the color component, through gray, to a complement of the color component may be displayed in association with the displayed function curve to suggest a result of color correction to be applied to the input image by manipulation of the function curve. A background image may suggest to the user what will happen if a point on the curve is moved. For example, if a user takes a point on the green curve that is in the center of the display and moves it to the lower right, the resulting image will have a magenta tint. The background image for the curve for the green channel shows more magenta in the lower right. A user can determine from an image to be corrected and the background image of the curves both which curve to use and approximately which direction to manipulate the curve.

Graphical user interface for color correction using curves

A video camera with an adaptive optic device, digital images, and a sequential diversity processor can reduce the optical aberrations introduced by a changing optical medium so as to produce sharper clarified images. The change in the optics between sequential video frames is diversity information which allows the sequential diversity processor to estimate both the object under observation and the aberration. No additional information, such as a defocused image or other sensing device, is required. The concept could be used in any video camera which outputs digital images and uses a digital processor to control the adaptive optic device between sequential frames.

Video camera utilizing sequential diversity imaging for image clarification

An ellipsoid chroma region of a chroma plane may be defined to specify a range of chromas for which a secondary chroma modification is to be applied, and to determine an output chroma for each chroma of the chroma plane based on the ellipsoid chroma region. Further, two or more different-shaped chroma regions of a chroma plane may be defined and manipulated, each chroma region specifying a range of chroma for which a secondary chroma modification is to be applied, and to determine an output chroma for each chroma of the chroma plane based on the two or more different-shaped chroma regions. Also, chroma-matching techniques may be used to define and manipulate the chroma region of a chroma plane that specifies a range of chroma for which a secondary chroma modification is to be applied, and an output chroma may be determined for each chroma of the chroma plane based on the chroma region.

Secondary color modification of a digital image

Two images are analyzed to compute a set of motion vectors that describes motion between the first and second images. A motion vector is computed for each pixel in an image at a time between the first and second images. This set of motion vectors may be defined at any time between the first and second images, such as the midpoint. The motion vectors may be computed using any of several techniques. An example technique is based on the constant brightness constraint, also referred to as optical flow. Each vector is specified at a pixel center in an image defined at the time between the first and second images. The vectors may point to points in the first and second images that are not on pixel centers. The motion vectors are used to warp the first and second images to a point in time of an output image between the first and second images using a factor that represents the time between the first and second image at which the output image occurs. The warped images are then blended using this factor to obtain the output image at the desired point in time between the first and second images. The point in time at which the output image occurs may be different from the time at which the motion vectors are determined. The same motion vectors may be used to determine two or more output images at different times between the first and second images. The images may be warped using a technique in which many small triangles are defined in an image corresponding in time to the point in time between the first and second images at which the motion vectors are determined. A transform for each small triangle from the point in time at which the motion vectors are determined to the desired interpolated image time is determined, e.g., the triangle is warped using the motion vectors associated with its vertices. For each pixel in each triangle in the output image, corresponding points in the first and second images are determined, and the first and second images are spatially sampled at these points. These samples for each pixel are combined to produce a value for that pixel in the output image.

Interpolation of a sequence of images using motion analysis

A graphical user interface facilitates color modification of a sequence of segments of moving images on a display of a general purpose computer. The graphical user interface may include a three-image display. The three image display may include a first region on the display for displaying an image from a current segment in the sequence to which a color modification is to be applied, a second region on the display and adjacent to the first region for displaying an image from a previous segment in the sequence before the current segment, and a third region on the display and adjacent to the first region for displaying an image from a next segment in the sequence after the current segment. A function screen allows a user to select a color modification to be performed to the current image. The function screen may include an interface that simultaneously displays a plurality of user modifiable graphs. The graphs may include a first graph for a red component, a second graph for a blue component, and a third graph for a green component. Each graph maps input values for a color component of a pixel on a first axis to output values for the color component for the pixel on a second axis. The graphical user interface also may include a color matcher.

Growing community of inventors

Wellesley, MA, United States of America

Robert A Gonsalves