Adobe, Inc.

Princeton University

Gautham J Mysore

Paris Smaragdis

Brian John King

Minje Kim

Matthew Douglas Hoffman

Jingwan Lu

Adam Finkelstein

Zeyu Jin

Peter Merrill

Cheng-i Wang

Dawen Liang

Dennis L Sun

Oliver Wang

Stephen J DiVerdi

Yang Zhang

Hijung Shin

Bryan Russell

Nicholas J Bryan

Bryan A Pardo

Francois Georges Germain

Zhengshan Shi

Shrikant Venkataramani

Bernd Huber

Ramin Anushiravani

Zhiyao Duan

Floraine Berthouzoz

Nicolas Maurice Boulanger-Lewandowski

François G Germain

Prem Seetharaman

Juhan Nam

Embodiments of the present invention provide systems, methods, and computer storage media for sound quality prediction and real-time feedback about sound quality, such as room acoustics quality and background noise. Audio data can be sampled from a live sound source and stored in an audio buffer. The audio data in the buffer is analyzed to calculate a stream of values of one or more sound quality measures, such as speech transmission index and signal-to-noise ratio. Speech transmission index can be calculated using a convolution neural network configured to predict speech transmission index from reverberant speech. The stream of values can be used to provide real-time feedback about sound quality of the audio data. For example, a visual indicator on a graphical user interface can be updated based on consistency of the values over time. The real-time feedback about sound quality can help users optimize their recording setup.

Sound quality prediction and interface to facilitate high-quality voice recordings

Certain embodiments involve transcript-based techniques for facilitating insertion of secondary video content into primary video content. For instance, a video editor presents a video editing interface having a primary video section displaying a primary video, a text-based navigation section having navigable portions of a primary video transcript, and a secondary video menu section displaying candidate secondary videos. In some embodiments, candidate secondary videos are obtained by using target terms detected in the transcript to query a remote data source for the candidate secondary videos. In embodiments involving video insertion, the video editor identifies a portion of the primary video corresponding to a portion of the transcript selected within the text-based navigation section. The video editor inserts a secondary video, which is selected from the candidate secondary videos based on an input received at the secondary video menu section, at the identified portion of the primary video.

Transcript-based insertion of secondary video content into primary video content

Techniques for a recursive deep-learning approach for performing speech synthesis using a repeatable structure that splits an input tensor into a left half and right half similar to the operation of the Fast Fourier Transform, performs a 1-D convolution on each respective half, performs a summation and then applies a post-processing function. The repeatable structure may be utilized in a series configuration to operate as a vocoder or perform other speech processing functions.

Real-time speaker-dependent neural vocoder

Time interval sound alignment techniques are described. In one or more implementations, one or more inputs are received via interaction with a user interface that indicate that a first time interval in a first representation of sound data generated from a first sound signal corresponds to a second time interval in a second representation of sound data generated from a second sound signal. A stretch value is calculated based on an amount of time represented in the first and second time intervals, respectively. Aligned sound data is generated from the sound data for the first and second time intervals based on the calculated stretch value.

Time interval sound alignment

Methods and systems for automatic audio loop generation from an audio track identify suitable portions of the audio track for generating audio loops. One or more embodiments identify portions of the audio track that include a beginning beat and an ending beat that have similar audio features that provide for seamless transitions when generating the audio loops. One or more embodiments generate scores for the portions based on the similarity of the audio features of the corresponding beginning and ending beats. Additionally, one or more embodiments use the generated scores to determine whether each portion is a suitable audio loop candidate. One or more embodiments then generate one or more audio loops using one or more suitable portions of the audio track.

Generating audio loops from an audio track

In some aspects, errors are replaced within an audio file by receiving a first audio sequence and a second audio sequence. The first audio sequence includes an erroneous subsequence and the second audio sequence includes a corrected subsequence for inclusion in the first audio sequence to replace the erroneous subsequence. The location of the erroneous subsequence in the first audio sequence is determined by applying a suitable matching operation (e.g., dynamic time warping). One or more matching subsequences of the first audio sequence located proximate to the erroneous subsequence in the first audio sequence and matching corresponding subsequences of the second audio sequence are located proximate to the corrected subsequence. A corrected first audio sequence is generated by replacing the erroneous subsequence and a matching subsequence of the first audio sequence with the corrected subsequence and the matching corresponding subsequence of the second audio sequence.

Automatic voiceover correction system

Embodiments of the present invention relate to automatically identifying structures of a music stream. A segment structure may be generated that visually indicates repeating segments of a music stream. To generate a segment structure, a feature that corresponds to a music attribute from a waveform corresponding to the music stream is extracted from a waveform, such as an input signal. Utilizing a signal segmentation algorithm, such as a Variable Markov Oracle (VMO) algorithm, a symbolized signal, such as a VMO structure, is generated. From the symbolized signal, a matrix is generated. The matrix may be, for instance, a VMO-SSM. A segment structure is then generated from the matrix. The segment structure illustrates a segmentation of the music stream and the segments that are repetitive.

Intuitive music visualization using efficient structural segmentation

Systems and techniques are disclosed for synthesizing a new word or short phrase such that it blends seamlessly in the context of insertion or replacement in an existing narration. In one such embodiment, a text-to-speech synthesizer is utilized to say the word or phrase in a generic voice. Voice conversion is then performed on the generic voice to convert it into a voice that matches the narration. An editor and interface are described that support fully automatic synthesis, selection among a candidate set of alternative pronunciations, fine control over edit placements and pitch profiles, and guidance by the editors own voice.

Text-based insertion and replacement in audio narration

Variable sound decomposition masking techniques are described. In one or more implementations, a mask is generated that incorporates a user input as part of the mask, the user input is usable at least in part to define a threshold that is variable based on the user input and configured for use in performing a sound decomposition process. The sound decomposition process is performed using the mask to assign portions of sound data to respective ones of a plurality of sources of the sound data.

Variable sound decomposition masks

Sound rate modification techniques are described. In one or more implementations, an indication is received of an amount that a rate of output of sound data is to be modified. One or more sound rate rules are applied to the sound data that, along with the received indication, are usable to calculate different rates at which different portions of the sound data are to be modified, respectively. The sound data is then output such that the calculated rates are applied.

Sound rate modification

Sound processing using a product-of-filters model is described. In one or more implementations, a model is formed by one or more computing devices for a time frame of sound data as a product of filters. The model is utilized by the one or more computing devices to perform one or more sound processing techniques on the time frame of the sound data.

Sound processing using a product-of-filters model

Embodiments of the present invention relate to enhancing sound through reverberation matching. In sonic implementations, a first sound recording recorded in a first environment is received. The first sound recording is decomposed to a first clean signal and a first reverb kernel. A second reverb kernel corresponding with a second sound recording recorded in a second environment is accessed, for example, based on a user indication to enhance the first sound recording to sound as though recorded in the second environment. An enhanced sound recording is generated based on the first clean signal and the second reverb kernel. The enhanced sound recording is a modification of the first sound recording to sound as though recorded in the second environment.

Sound enhancement through reverberation matching

Pattern identification using convolution is described. In one or more implementations, a representation of a pattern is obtained that is described using data points that include frequency coordinates, time coordinates, and energy values. An identification is made as to whether sound data described using irregularly positioned data points includes the pattern, the identifying including use of a convolution of the frequency or time coordinates to determine correspondence with the representation of the pattern.

Irregular pattern identification using landmark based convolution

Online source separation may include receiving a sound mixture that includes first audio data from a first source and second audio data from a second source. Online source separation may further include receiving pre-computed reference data corresponding to the first source. Online source separation may also include performing online separation of the second audio data from the first audio data based on the pre-computed reference data.

Growing community of inventors

San Francisco, CA, United States of America

Gautham J Mysore