Smule, Inc.

Other

Nick Kruge

Ge Wang

Perry Raymond Cook

Mark T Godfrey

Robert Hamilton

Turner Evan Kirk

Ian S Simon

A generally diverse set of audiovisual clips is sourced from one or more repositories for use in preparing a coordinated audiovisual work. In some cases, audiovisual clips are retrieved using tags such as user-assigned hashtags or metadata. Pre-existing associations of such tags can be used as hints that certain audiovisual clips are likely to share correspondence with an audio signal encoding of a particular song or other audio baseline. Clips are evaluated for computationally determined correspondence with an audio baseline track. In general, comparisons of audio power spectra, of rhythmic features, tempo, pitch sequences and other extracted audio features may be used to establish correspondence. For clips exhibiting a desired level of correspondence, computationally determined temporal alignments of individual clips with the baseline audio track are used to prepare a coordinated audiovisual work that mixes the selected audiovisual clips with the audio track.

Coordinated audiovisual montage from selected crowd-sourced content with alignment to audio baseline

Despite practical limitations imposed by mobile device platforms and applications, truly captivating musical instruments may be synthesized in ways that allow musically expressive performances to be captured and rendered in real-time. Visual cues presented on a multi-touch sensitive display provide the user with temporally sequenced string excitation cues. Note or chord soundings are indicated by user gestures (e.g., pluck-type gestures, strum-type gestures, chord selections, etc.) captured at the multi-touch sensitive display. Those captured gestures, rather than simply the score itself, are used as inputs to a digital synthesis of the musical instrument.

Score-directed string retuning and gesture cueing in synthetic multi-string musical instrument

Building on a set of captured audiovisual segments clipped to provide a palette of audio and synched video, techniques and implementations described herein facilitate a new and highly-personalized (and in some cases crowd-sourced or sourcable) genre of audiovisual sampling and musical performance. Using a palette of captured and/or imported audio and associated video, users can remix to create a coordinated audiovisual performance. Because the audiovisual sampling and musical performance capabilities can be hosted on ubiquitous handheld or other portable computing devices such as smartphones and/or pad-type computers, user/musicians can, in essence, creatively remix their life.

Audiovisual sampling for percussion-type instrument with crowd-sourced content sourcing and distribution

A synthetic multi-string musical instrument captures a stream of expressive gestures indicated on a multi-touch sensitive display for note/chord soundings and associated performance effects and embellishments. Visual cues in accord with a musical score may be revealed/advanced at a current performance tempo, but it is the user's gestures that actually drive the audible performance rendering via digital synthesis. Opportunities for user expression (or variance from score) include onset and duration of note soundings, tempo changes, as well as uncued string bend effects, vibrato, etc. Gesturing mechanism are provide to allow user musicians to sound chords without having to register precisely accurate multi-touch screen contacts. This can be especially helpful for mobile phone, media player and game controller embodiments, where there is generally limited real-estate to display six (6) or more strings, and user fingers are generally too fat to precisely contact such strings.

Growing community of inventors

San Francisco, CA, United States of America

Nick Kruge