Avid Technology Limited

Randy M Fayan

Katherine H Cornog

Garth Allen Dickie

Peter J Fasciano

Robert A Gonsalves

Michael D Laird

Playback of a graphical representation of a digital musical score is synchronized with an expressive audio rendering of the score that contains tempo and dynamics beyond those specified in the score. The method involves determining a set of offsets for occurrences of score events in the audio rendering by comparing and temporally aligning audio waveforms of successive subclips of the audio rendering with corresponding audio waveforms of successive subclips of an audio rendering synthesized directly from the score. Tempos and dynamics of human performances may be extracted and used to generate expressive renderings synthesized from the corresponding digital score. This enables parties who wish to distribute or share music scores, such as composers and publishers, to allow prospective licensors to evaluate the score by listening to an expressive musical recording instead of a mechanically synthesized rendering.

Synchronizing playback of a digital musical score with an audio recording

In calculating motion between two images, a single channel image may be generated for each image based on measurement of a desired characteristic of those images. Given a desired characteristic (such as edge strength or edge magnitude) in an image, a function measures the strength of the desired characteristic in a region around a pixel in an image. A range of values can represent the likelihood, or measure of confidence, of the occurrence of the desired characteristic in the region around the pixel. Thus, each pixel in the single channel image has a value from the range of values that is determined according to a function. This function operates on a neighborhood in the input image that corresponds to the pixel in the single channel image, and measures the likelihood of occurrence of, or strength of, the desired characteristic in that neighborhood. Two single channel images generated from two input images are analyzed to provide a motion estimate that indicates how the location of characteristics in the images changes from one image to the next image. If the desired characteristic is an edge magnitude or edge strength, then the motion is effectively estimated using a constant edge constraint.

Analyzing motion of characteristics in images

A keyer is provided with a graphical user interface that helps a user visualize the relationship between the key and the image to be processed using that key. A color space swatch is processed by the keyer using the defined key. The output of the keyer as applied to the color space swatch is displayed to the user to illustrate which colors match the defined key. The alpha matte generated by applying the key to the color space swatch also may be displayed. Each pixel in either the color space swatch or the alpha matte generated by applying the key to the color space swatch may be modified to indicate whether its corresponding color is present in the input image or in the preprocessed input image. Luminance processing also may be applied. In particular, the alpha value for a pixel may be adjusted according to the luminance of the pixel according to a user defined function. The alpha matte displayed to the user may include the effects of such luminance processing. The keyer may preprocess the input image by modifying the saturation of each pixel using the luminance of the pixel. These displays may be interactively updated as the user modifies the definition of the key or the controls for preprocessing or luma processing.

Graphical user interface for a keyer

A retiming function that defines a rampable retiming effect is used to generate new audio and video samples at appropriate output times. In particular, for each output time, a corresponding input time is determined from the output time by using the retiming function. The retiming function may be a speed curve, a position curve that maps output times to input times directly or a mapping defining correspondence times between points in the video data and points in the audio data. An output sample is computed for the output time based on at least the data in the neighborhood of the corresponding input time, using a resampling function for the type of media data. Synchronization is achieved by ensuring that the input times determined to correspond to output times for video samples correspond to the input times determined to correspond to the same output times for audio samples.

Performance retiming effects on synchronized data in an editing system

Two images are analyzed to compute a set of motion vectors that describes motion between the first and second images. A motion vector is computed for each pixel in an image at a time between the first and second images. This set of motion vectors may be defined at any time between the first and second images, such as the midpoint. The motion vectors may be computed using any of several techniques. An example technique is based on the constant brightness constraint, also referred to as optical flow. Each vector is specified at a pixel center in an image defined at the time between the first and second images. The vectors may point to points in the first and second images that are not on pixel centers. The motion vectors are used to warp the first and second images to a point in time of an output image between the first and second images using a factor that represents the time between the first and second image at which the output image occurs. The warped images are then blended using this factor to obtain the output image at the desired point in time between the first and second images. The point in time at which the output image occurs may be different from the time at which the motion vectors are determined. The same motion vectors may be used to determine two or more output images at different times between the first and second images. The images may be warped using a technique in which many small triangles are defined in an image corresponding in time to the point in time between the first and second images at which the motion vectors are determined. A transform for each small triangle from the point in time at which the motion vectors are determined to the desired interpolated image time is determined, e.g., the triangle is warped using the motion vectors associated with its vertices. For each pixel in each triangle in the output image, corresponding points in the first and second images are determined, and the first and second images are spatially sampled at these points. These samples for each pixel are combined to produce a value for that pixel in the output image.

Interpolation of a sequence of images using motion analysis

Visibie artifacts in images created using image processing based on motion vector maps may be reduced by providing one or more mechanisms for correcting the vector map. In general, the set of motion vectors is changed by selecting one or more portions of the image. The vectors corresponding to the selected one or more portions are modified. Various image processing operations, such as motion compensated interpolation, may be performed using the changed set of motion vectors.

Correcting motion vector maps for image processing

Two images are analyzed to compute a set of motion vectors that describes motion between the first and second images. A motion vector is computed for each pixel in an image at a time between the first and second images. This set of motion vectors may be defined at any time between the first and second images, such as the midpoint. The motion vectors may be computed using any of several techniques. An example technique is based on the constant brightness constraint, also referred to as optical flow. Each vector is specified at a pixel center in an image defined at the time between the first and second images. The vectors may point to points in the first and second images that are not on pixel centers. The motion vectors are used to warp the first and second images to a point in time of an output image between the first and second images using a factor that represents the time between the first and second image at which the output image occurs. The warped images are then blended using this factor to obtain the output image at the desired point in time between the first and second images. The point in time at which the output image occurs may be different from the time at which the motion vectors are determined. The same motion vectors may be used to determine two or more output images at different times between the first and second images. The images may be warped using a technique in which many small triangles are defined in an image corresponding in time to the point in time between the first and second images at which the motion vectors are determined. A transform for each small triangle from the point in time at which the motion vectors are determined to the desired interpolated image time is determined, e.g., the triangle is warped using the motion vectors associated with its vertices. For each pixel in each triangle in the output image, corresponding points in the first and second images are determined, and the first and second images are spatially sampled at these points. These samples for each pixel are combined to produce a value for that pixel in the output image. A motion vector map also may be generated between two fields of opposite field sense. An offset introduced by such calculation may be removed. During warping or other operation using the adjusted vector map, when sampling a field of one field sense to generate a field of an opposite field sense, that offset is reintroduced by translating the sampling region.

Growing community of inventors

Medford, MA, United States of America

Randy M Fayan