Google Inc.

The Echo Nest Corporation

Eastman-Kodak Company

Columbia University

International Computer Science Institute

Daniel Ellis

Ryan Michael Rifkin

Aren Jansen

Brian Whitman

Alexander C Loui

Shih-Fu Chang

Sangita R Sharma

Katherine Irene Chou

Alvin Rishi Rajkomar

Sourish Chaudhuri

Hynek Hermansky

Wei Jiang

Henry George Wei

Manoj Plakal

David Parish

Courtenay Cotton

Shawn Hershey

Michael Dwight Howell

David Michael Parish

Kasumi Widner

Michael Philip Brenner

Andrew Nesbit

Nathan Reale

Fangzhou Wang

Jiayang Liu

Ratheet Pandya

Richard Channing Moore

The present disclosure provides systems and methods that generating health diagnostic information from an audio recording. A computing system can include a machine-learned health model comprising that includes a sound model trained to receive data descriptive of a patient audio recording and output sound description data. The computing system can include a diagnostic model trained to receive the sound description data and output a diagnostic score. The computing system can include at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed, cause the processor to perform operations. The operations can include obtaining the patient audio recording; inputting data descriptive of the patient audio recording into the sound model; receiving, as an output of the sound model, the sound description data; inputting the sound description data into the diagnostic model; and receiving, as an output of the diagnostic model, the diagnostic score.

System and method for generating diagnostic health information using deep learning and sound understanding

A computing system can include an embedding model and a clustering model. The computing system input each of the plurality of inputs into the embedding model and receiving respective embeddings for the plurality of inputs as outputs of the embedding model. The computing system can input the respective embeddings for the plurality of inputs into the clustering model and receiving respective cluster assignments for the plurality of inputs as outputs of the clustering model. The computing system can evaluate a clustering loss function that evaluates a first average, across the plurality of inputs, of a respective first entropy of each respective probability distribution; and a second entropy of a second average of the probability distributions for the plurality of inputs. The computing system can modify parameter(s) of one or both of the clustering model and the embedding model based on the clustering loss function.

Minimum-example/maximum-batch entropy-based clustering with neural networks

Methods are provided for generating training triplets that can be used to train multidimensional embeddings to represent the semantic content of non-speech sounds present in a corpus of audio recordings. These training triplets can be used with a triplet loss function to train the multidimensional embeddings such that the embeddings can be used to cluster the contents of a corpus of audio recordings, to facilitate a query-by-example lookup from the corpus, to allow a small number of manually-labeled audio recordings to be generalized, or to facilitate some other audio classification task. The triplet sampling methods may be used individually or collectively, and each represent a respective heuristic about the semantic structure of audio recordings.

Unsupervised learning of semantic audio representations

A content server accessing an audio stream, and inputs portions of the audio stream into one or more non-speech classifiers for classification, the non-speech classifiers generating, for portions of the audio stream, a set of raw scores representing likelihoods that the respective portion of the audio stream includes an occurrence of a particular class of non-speech sounds associated with each of the non-speech classifiers. The content server generates binary scores for the sets of raw scores, the binary scores generated based on a smoothing of a respective set of raw scores. The content server applies a set of non-speech captions to portions of the audio stream in time, each of the sets of non-speech captions based on a different one of the set binary scores of the corresponding portion of the audio stream.

Automatic smoothed captioning of non-speech sounds from audio

Methods, computing devices, and machine readable storage media for generating a fingerprint of a music sample. The music sample may be filtered into a plurality of frequency bands. Onsets in each of the frequency bands may be independently detected. Inter-onset intervals between pairs of onsets within the same frequency band may be determined. At least one code associated with each onset may be generated, each code comprising a frequency band identifier identifying a frequency band in which the associated onset occurred and one or more inter-onset intervals. Each code may be associated with a timestamp indicating when the associated onset occurred within the music sample. All generated codes and the associated timestamps may be combined to form the fingerprint.

Musical fingerprinting based on onset intervals

A method for fingerprinting an unknown music sample is disclosed. A plurality of known tracks may be segmented into reference samples. A reference fingerprint including a plurality of codes may be generated for each reference sample. An inverted index including, for each possible code value, a list of reference samples having reference fingerprints that contain the respective code value may be generated. An unknown fingerprint including a plurality of codes may be generated from the unknown music sample. A code match histogram may list candidate reference samples and associated scores, each score indicating a number of codes from the unknown fingerprint that match codes in the reference fingerprint. Time difference histograms may be generated for two or more reference samples having the highest scores. A determination may be made whether or not a single reference sample matches the unknown music sample based on a comparison of the time difference histograms.

Musical fingerprinting

A method for determining a classification for a video segment, comprising the steps of: breaking the video segment into a plurality of short-term video slices, each including a plurality of video frames and an audio signal; analyzing the video frames for each short-term video slice to form a plurality of region tracks; analyzing each region track to form a visual feature vector and a motion feature vector; analyzing the audio signal for each short-term video slice to determine an audio feature vector; forming a plurality of short-term audio-visual atoms for each short-term video slice by combining the visual feature vector and the motion feature vector for a particular region track with the corresponding audio feature vector; and using a classifier to determine a classification for the video segment responsive to the short-term audio-visual atoms.

Video concept classification using audio-visual atoms

The present invention successfully combines neural-net discriminative feature processing with Gaussian-mixture distribution modeling (GMM). By training one or more neural networks to generate subword probability posteriors, then using transformations of these estimates as the base features for a conventionally-trained Gaussian-mixture based system, substantial error rate reductions may be achieved. The present invention effectively has two acoustic models in tandem&#x2014;first a neural net and then a GMM. By using a variety of combination schemes available for connectionist models, various systems based upon multiple features streams can be constructed with even greater error rate reductions.

Nonlinear mapping for feature extraction in automatic speech recognition

Growing community of inventors

New York, NY, United States of America

Daniel Ellis