The University of North Carolina at Chapel Hill

University System of Maryland

Dinesh Manocha

Ravish Mehra

Lakulish Shailesh Antani

Carl Henry Schissler

Uttaran Bhattacharya

Trisha Mittal

Rohan Chandra

Aniket Bera

Micah Taylor

Anish Chandak

Jan-Michael Frahm

Enrique Dunn

Alok Namdeo Meshram

Hongsheng Yang

Nicolas Manuel Morales

Pooja Guhan

A video is classified as real or fake by extracting facial features, including facial modalities and facial emotions, and speech features, including speech modalities and speech emotions, from the video. The facial and speech modalities are passed through first and second neural networks, respectively, to generate facial and speech modality embeddings. The facial and speech emotions are passed through third and fourth neural networks, respectively, to generate facial and speech emotion embeddings. A first distance, d, between the facial modality embedding and the speech modality embedding is generated, together with a second distance, d, between the facial emotion embedding and the speech emotion embedding. The video is classified as fake if a sum of the first distance and the second distance exceeds a threshold distance. The networks may be trained using real and fake video pairs for multiple subjects.

System and method for detecting fabricated videos

Systems, methods, apparatuses, and computer program products for recognizing human emotion in images or video. A method for recognizing perceived human emotion may include receiving a raw input. The raw input may be processed to generate input data corresponding to at least one context. Features from the raw input data may be extracted to obtain a plurality of feature vectors and inputs. The plurality of feature vectors and the inputs may be transmitted to a respective neural network. At least some of the plurality of feature vectors may be fused to obtain a feature encoding. Additional feature encodings may be computed from the plurality of feature vectors via the respective neural network. A multi-label emotion classification of a primary agent may be performed in the raw input based on the feature encoding and the additional feature encodings.

Human emotion recognition in images or video

Systems, methods, apparatuses, and computer program products for providing multimodal emotion recognition. The method may include receiving raw input from an input source. The method may also include extracting one or more feature vectors from the raw input. The method may further include determining an effectiveness of the one or more feature vectors. Further, the method may include performing, based on the determination, multiplicative fusion processing on the one or more feature vectors. The method may also include predicting, based on results of the multiplicative fusion processing, one or more emotions of the input source.

System and method for multimodal emotion recognition

Methods, systems, and computer readable media for simulating sound propagation are disclosed. According to one method, the method includes decomposing a virtual environment scene including at least one object into a plurality of surface regions, wherein each of the surface regions includes a plurality of surface patches. The method further includes organizing sound rays generated by a sound source in the virtual environment scene into a plurality of path tracing groups, wherein each of the path tracing groups comprises a group of the rays that traverses a sequence of surface patches. The method also includes determining, for each of the path tracing groups, a sound intensity by combining a sound intensity computed for a current time with one or more previously computed sound intensities respectively associated with previous times and generating a simulated output sound at a listener position using the determined sound intensities.

Methods, systems, and computer readable media for modeling interactive diffuse reflections and higher-order diffraction in virtual environment scenes

Methods, systems, and computer readable media for acoustic classification and optimization for multi-modal rendering of real-world scenes are disclosed. According to one method for determining acoustic material properties associated with a real-world scene, the method comprises obtaining an acoustic response in a real-world scene. The method also includes generating a three-dimensional (3D) virtual model of the real-world scene. The method further includes determining acoustic material properties of surfaces in the 3D virtual model using a visual material classification algorithm to identify materials in the real-world scene that make up the surfaces and known acoustic material properties of the materials. The method also includes using the acoustic response in the real-world scene to adjust the acoustic material properties.

Methods, systems, and computer readable media for acoustic classification and optimization for multi-modal rendering of real-world scenes

Methods, systems, and computer readable media for conducting interactive sound propagation and rending for a plurality of sound sources in a virtual environment scene are disclosed. According to one method, the method includes decomposing a virtual environment scene containing a plurality of sound sources into a plurality of partitions and forming a plurality of source group clusters, wherein each of the source group clusters includes two or more of the sound sources located within a common partition. The method further includes determining, for each of the source group clusters, a single set of sound propagation paths relative to a listener position and generating a simulated output sound at a listener position using sound intensities associated with the determined sets of sound propagation paths.

Methods, systems, and computer readable media for conducting interactive sound propagation and rendering for a plurality of sound sources in a virtual environment scene

Methods, systems, and computer readable media for utilizing ray-parameterized reverberation filters to facilitate interactive sound rendering are disclosed. According to one method, the method includes generating a sound propagation impulse response characterized by a plurality of predefined number of frequency bands and estimating a plurality of reverberation parameters for each of the predefined number of frequency bands of the impulse response. The method further includes utilizing the reverberation parameters to parameterize a plurality of reverberation filters in an artificial reverberator, rendering an audio output in a spherical harmonic (SH) domain that results from a mixing of a source audio and a reverberation signal that is produced from the artificial reverberator, and performing spatialization processing on the audio output.

Methods, systems, and computer readable media for utilizing ray-parameterized reverberation filters to facilitate interactive sound rendering

Methods, systems, and computer readable media for utilizing adaptive rectangular decomposition (ARD) to perform head-related transfer function (HRTF) simulations are disclosed herein. According to one method, the method includes obtaining a mesh model representative of head and ear geometry of a listener entity and segmenting a simulation domain of the mesh model into a plurality of partitions. The method further includes conducting an ARD simulation on the plurality of partitions to generate simulated sound pressure signals within each of the plurality of partitions and processing the simulated sound pressure signals to generate at least one HRTF that is customized for the listener entity.

Methods, systems, and computer readable media for utilizing adaptive rectangular decomposition (ARD) to generate head-related transfer functions

Methods, systems, and computer readable media for utilizing parallel adaptive rectangular decomposition (ARD) to perform acoustic simulations are disclosed herein. According to one method, the method includes assigning, to each of a plurality of processors in a central processing unit (CPU) cluster, ARD processing responsibilities associated with one or more of a plurality of partitions of an acoustic space and determining, by each processor, pressure field data corresponding to the one or more assigned partitions. The method further includes transferring, by each processor, the pressure field data to at least one remote processor that is assigned to a partition that shares an interface with at least one partition assigned to the transferring processor and receiving, by each processor from the at least one remote processor, forcing term values that have been derived by the at least one remote processor using the pressure field data.

Methods, systems, and computer readable media for utilizing parallel adaptive rectangular decomposition (ARD) to perform acoustic simulations

The subject matter described herein includes an approach for wave-based sound propagation suitable for large, open spaces spanning hundreds of meters, with a small memory footprint. The scene is decomposed into disjoint rigid objects. The free-field acoustic behavior of each object is captured by a compact per-object transfer-function relating the amplitudes of a set of incoming equivalent sources to outgoing equivalent sources. Pairwise acoustic interactions between objects are cornuted analytically, yielding compact inter-object transfer functions. The global sound field accounting for all orders of interaction is computed using these transfer functions. The runtime system uses fast summation over the outgoing equivalent source amplitudes for all objects to auralize the sound field at a moving listener in real-time. We demonstrate realistic acoustic effects such as diffraction, low-passed sound behind obstructions, focusing, scattering, high-order reflections, and echoes, on a variety of scenes.

Methods, systems, and computer readable media for simulating sound propagation in large scenes using equivalent sources

Methods, systems, and computer readable media for supporting source or listener directivity in a wave-based sound propagation model are disclosed. According to one method, the method includes computing, prior to run-time, one or more sound fields associated with a source or listener position and modeling, at run-time and using the one or more sound fields and a wave-based sound propagation model, source or listener directivity in an environment.

Methods, systems, and computer readable media for source and listener directivity for interactive wave-based sound propagation

The subject matter described herein includes a method for simulating directional sound reverberation. The method includes performing ray tracing from a listener position in a scene to surface as visible from a listener position. The method further includes determining a directional local visibility representing a distance from a listener position to nearer surface in the scene alone each ray. The method further includes determining directional reverberation at the listener position based on the directional local visibility. The method further includes rendering a simulated sound indicative of the directional reverberation at the listener position.

Aural proxies and directionally-varying reverberation for interactive sound propagation in virtual environments

Methods, systems, and computer program products for simulating propagation of sound in a static scene can be operated for pre-computing a transfer operator for simulating results of sound reflection within a modeled scene, simulating distribution of sound energy from a sound source positioned within the scene, applying the transfer operator to the distribution of sound energy from the sound source to compute simulated sound at a plurality of points within the scene, and simulating collection of sound energy from the plurality of points within the scene to a listener positioned within the scene.

Methods and systems for direct-to-indirect acoustic radiance transfer

Methods, systems, and computer program products for simulating sound propagation can be operable to define a sound source position within a modeled scene having a given geometry and construct a visibility tree for modeling sound propagation paths within the scene. Using from-region visibility techniques to model sound diffraction and from-point visibility technique to model specular sound reflections within the scene, the size of the visibility tree can be reduced. Using the visibility tree, an impulse response can be generated for the scene, and the impulse response can be used to simulate sound propagation in the scene.

Growing community of inventors

Chapel Hill, NC, United States of America

Dinesh Manocha