Adobe, Inc.

Princeton University

University of Washington

Adam Finkelstein

Zeyu Jin

Elya Shechtman

Jingwan Lu

Connelly Barnes

Stephen J DiVerdi

Gautham J Mysore

Hijung Shin

David H Salesin

Wilmot Li

Dan Goldman

Jiaqi Su

Nora Willett

Daniel Goldman

Charles Edward Jacobs

Jacob Lewellen

Embodiments are disclosed for converting audio data to studio quality audio data. The method includes obtaining an audio data having a first quality for conversion to studio quality audio. A first machine learning model predicts a set of acoustic features. A spectral mask is applied to the audio data during the prediction of the set of acoustic features. A second machine learning model generates studio quality audio from the set of acoustic features and the audio data.

Studio quality audio enhancement

Embodiments are disclosed for generating full-band audio from narrowband audio using a GAN-based audio super resolution model. A method of generating full-band audio may include receiving narrow-band input audio data, upsampling the narrow-band input audio data to generate upsampled audio data, providing the upsampled audio data to an audio super resolution model, the audio super resolution model trained to perform bandwidth expansion from narrow-band to wide-band, and returning wide-band output audio data corresponding to the narrow-band input audio data.

High fidelity audio super resolution

Operations of a method include receiving a request to enhance a new source audio. Responsive to the request, the new source audio is input into a prediction model that was previously trained. Training the prediction model includes providing a generative adversarial network including the prediction model and a discriminator. Training data is obtained including tuples of source audios and target audios, each tuple including a source audio and a corresponding target audio. During training, the prediction model generates predicted audios based on the source audios. Training further includes applying a loss function to the predicted audios and the target audios, where the loss function incorporates a combination of a spectrogram loss and an adversarial loss. The prediction model is updated to optimize that loss function. After training, based on the new source audio, the prediction model generates a new predicted audio as an enhanced version of the new source audio.

Using a predictive model to automatically enhance audio having various audio quality issues

This disclosure generally relates to character animation. More specifically, this disclosure relates to pose selection using data analytics techniques applied to training data, and generating 2D animations of illustrated characters using performance data and the selected poses. An example process or system includes extracting sets of joint positions from a training video including the subject, grouping the plurality of frames into frame groups using the sets of joint positions for each frame, identifying a representative frame for each frame group using the frame groups, clustering the frame groups into clusters using the representative frames, outputting a visualization of the clusters at a user interface, and receiving a selection of a cluster for animation of the subject.

Pose selection and animation of characters using video data and training techniques

This disclosure generally relates to character animation. More specifically, but not by way of limitation, this disclosure relates to pose selection using data analytics techniques applied to training data, and generating 2D animations of illustrated characters using performance data and the selected poses. An example process or system includes obtaining a selection of training poses of the subject and a set of character poses, obtaining a performance video of the subject, wherein the performance video includes a plurality of performance frames that include poses performed by the subject, grouping the plurality of performance frames into groups of performance frames, assigning a selected training pose from the selection of training poses to each group of performance frames using the clusters of training frames, generating a sequence of character poses based on the groups of performance frames and their assigned training poses, outputting the sequence of character poses.

Techniques for a recursive deep-learning approach for performing speech synthesis using a repeatable structure that splits an input tensor into a left half and right half similar to the operation of the Fast Fourier Transform, performs a 1-D convolution on each respective half, performs a summation and then applies a post-processing function. The repeatable structure may be utilized in a series configuration to operate as a vocoder or perform other speech processing functions.

Real-time speaker-dependent neural vocoder

Systems and techniques are disclosed for synthesizing a new word or short phrase such that it blends seamlessly in the context of insertion or replacement in an existing narration. In one such embodiment, a text-to-speech synthesizer is utilized to say the word or phrase in a generic voice. Voice conversion is then performed on the generic voice to convert it into a voice that matches the narration. An editor and interface are described that support fully automatic synthesis, selection among a candidate set of alternative pronunciations, fine control over edit placements and pitch profiles, and guidance by the editors own voice.

Text-based insertion and replacement in audio narration

Systems and methods for synthesizing paintbrush strokes may use high fidelity pose data of reference strokes to supplement lower dimension stroke data. For example, 6DOF data representing reference strokes created by skilled artists may be captured and stored in a library of reference strokes. A query stroke made by a less skilled artist/user and/or made using input devices that do not provide 6DOF data may be processed by a stroke synthesis engine to produce an output stroke that follows the trajectory of the query stroke and includes pose data from one or more reference strokes. The stroke synthesis engine may construct feature vectors for samples of reference strokes and query strokes, select the best neighbor feature vector from the library for each query stroke sample, assign the pose data of the best neighbor to the query sample, and smooth the sequence of assigned poses to produce the synthesized output stroke.

Systems and methods for synthesizing high fidelity stroke data for lower dimension input strokes

Techniques for determining correspondence between image regions are described. A computing system stores images that are comparable to determine corresponding image patches of the images. An approximation algorithm is implemented, and for multiple image patches in a region in a first image, corresponding image patches are determined in a second image. The approximation algorithm performs iterations utilizing a nearby-pixel mapping evaluation and a random-perturbation mapping evaluation to determine and select the corresponding image patches in the second image.

Determining correspondence between image regions

A computer-implemented method for determining correspondence between images includes: receiving images in a computer system; performing iterations using the computer system to find respective mappings for each patch of pixels in the images to a patch in another one of the images such that the mappings have minimal patch distance, the iterations including at least: (i) evaluation of a nearby-pixel mapping in a current image, (ii) evaluation of a randomly selected mapping in the current image, and (iii) evaluation of a randomly selected mapping in another one of the images; and generating a mapping record that results from the iterations.

Image processing using image web

Determining correspondence between image regions can include: selecting first and second regions of visual content including pixels in a computer system, the first region comprising a first patch to be mapped to the second region; selecting at least two heuristics for use in mapping the first patch to the second region, the heuristics selected from the group consisting of: (i) nearby-pixel mapping evaluation; (ii) random-perturbation mapping evaluation; (iii) evaluation of multiple mapping candidates identified in an iterative search process; and (iv) enrichment to increase a collection of mapping candidates; and identifying, using the selected heuristics, at least one patch in the second region for the first patch.

A representation for encoding time varying image data that allows for varying spatial and temporal resolutions in different parts of a video sequence. The representation, called multiresolution video, is based on a sparse, hierarchical encoding of the video data as multiple streams. Operations are defined for creating, viewing, and editing multiresolution video sequences. These operations support a variety of applications, including multiresolution playback, motion-blurred &ldquo;fast forward&rdquo; and &ldquo;reverse,&rdquo; constant speed display, enhanced video shuttling or searching, and &ldquo;video clip-art&rdquo; editing and compositing. The multiresolution representation requires little storage overhead, and the algorithms using the representation are both simple and efficient.

Multiresolution video

A method and system for editing a figure is provided. The method performs the steps of storing a representation of the figure in digital format in a storage medium, separating the representation of the figure into a general sweep feature and a detail feature, and editing at least one of the general sweep feature and/or the detail feature to produce a processed general sweep feature and a processed detail feature. The processed general sweep feature and the processed detail feature are then combined into an edited representation of the figure, which is output from the storage medium.

Growing community of inventors

Princeton, NJ, United States of America

Adam Finkelstein