Massachusetts Institute of Technology

Quanta Computer Incorporated

Adobe, Inc.

Shell Oil Comapny

Qatar Foundation for Education, Science and Community Development

Frederic Durand

William T Freeman

Neal Wadhwa

Michael Rubinstein

John V Guttag

Myers Abraham Davis

Hao-yu Wu

Eugene Inghaw Shih

Wojciech Matusik

Justin Gejune Chen

Sylvain Philippe Paris

Eric Chan

Vladimir Leonid Bychkovsky

Pitchaya Sitthi-Amorn

Oral Buyukozturk

Tianfan Xue

Samuel William Hasinoff

Szu-Po Wang

Petr Kellnhofer

Piotr Krzysztof Didyk

Piotr K Didyk

Guha Balakrishnan

YiChang Shih

Mohamed Hefeeda

Tali Dekel

Theodericus Johannes Henricus Smit

John William Fisher, Iii

Mohamed Abdelaziz A Mohamed Elgharib

Hossein Mobahi

Zoran Dzunic

Donglai Wei

James Long

Reza Mohammadi Ghazi

Song-Pei Du

Sergio Daniel Kapusta

Alec Rivers

Heart rates and beat lengths of a subject can be extracted from videos of the subject by measuring subtle head motion caused by the Newtonian reaction to the influx of blood at each beat. Embodiments track features on the video images of the subject's head and perform principal component analysis (PCA) to decompose the feature location-time series into a set of component motions. The method or system then selects a component that best corresponds to heartbeats based on its temporal frequency spectrum. Finally, the motion projected to this component is analyzed and peaks of the location-time series are identified, which correspond to heartbeats. Pulse rate measurements or heart rate measurements of the subject are output.

Pulse detection from head motions in video

Structural health monitoring (SHM) is essential but can be expensive to perform. In an embodiment, a method includes sensing vibrations at a plurality of locations of a structure by a plurality of time-synchronized sensors. The method further includes determining a first set of dependencies of all sensors of the time-synchronized sensors at a first sample time to any sensors of a second sample time, and determining a second set of dependencies of all sensors of the time-synchronized sensors at the second sample time to any sensors of a third sample time. The second sample time is later than the first sample time, and the third sample time is later than the second sample time. The method then determines whether the structure has changed if the first set of dependencies is different from the second set of dependencies. Therefore, automated SHM can ensure safety at a lower cost to building owners.

Motion sensing wi-fi sensor networks for continuous 3D modeling and prediction of facility responses to disturbances

A method and system of converting stereo video content to multi-view video content combines an Eulerian approach with a Lagrangian approach. The method comprises generating a disparity map for each of the left and right views of a received stereoscopic frame. For each corresponding pair of left and right scanlines of the received stereoscopic frame, the method further comprises decomposing the left and right scanlines into a left sum of wavelets or other basis functions, and a right sum wavelets or other basis functions. The method further comprises establishing an initial disparity correspondence between left wavelets and right wavelets based on the generated disparity maps, and refining the initial disparity between the left wavelet and the right wavelet using a phase difference between the corresponding wavelets. The method further comprises reconstructing at least one novel view based on the left and right wavelets.

3DTV at home: Eulerian-Lagrangian stereo-to-multi-view conversion

Heart rates and beat lengths can be extracted from videos by measuring subtle head motion caused by the Newtonian reaction to the influx of blood at each beat. In an embodiment of the present invention, a method tracks features on the head and performs principal component analysis (PCA) to decompose their trajectories into a set of component motions. The method then selects a component that best corresponds to heartbeats based on its temporal frequency spectrum. Finally, the motion projected to this component is analyzed and peaks of the trajectories are identified, which correspond to heartbeats.

An apparatus according to an embodiment of the present invention enables measurement and visualization of a refractive field such as a fluid. An embodiment device obtains video captured by a video camera with an imaging plane. Representations of apparent motions in the video are correlated to determine actual motions of the refractive field. A textured background of the scene can be modeled as stationary, with a refractive field translating between background and video camera. This approach offers multiple advantages over conventional fluid flow visualization, including an ability to use ordinary video equipment outside a laboratory without particle injection. Even natural backgrounds can be used, and fluid motion can be distinguished from refraction changes. Embodiments can render refractive flow visualizations for augmented reality, wearable devices, and video microscopes.

Devices for refractive field visualization

A method and corresponding apparatus for measuring object motion using camera images may include measuring a global optical flow field of a scene. The scene may include target and reference objects captured in an image sequence. Motion of a camera used to capture the image sequence may be determined relative to the scene by measuring an apparent, sub-pixel motion of the reference object with respect to an imaging plane of the camera. Motion of the target object corrected for the camera motion may be calculated based on the optical flow field of the scene and on the apparent, sub-pixel motion of the reference object with respect to the imaging plane of the camera. Embodiments may enable measuring vibration of structures and objects from long distance in relatively uncontrolled settings, with or without accelerometers, with high signal-to-noise ratios.

Methods and devices for measuring object motion using camera images

Embodiments can be used to synthesize physically plausible animations of target objects responding to new, previously unseen forces. Knowledge of scene geometry or target material properties is not required, and a basis set for creating realistic synthesized motions can be developed using only input video of the target object. Embodiments can enable new animation and video production techniques.

Methods and apparatus for modeling deformations of an object

In one embodiment, a method comprises projecting, from a projector, a diffused on an object. The method further includes capturing, with a first camera in a particular location, a reference image of the object while the diffused is projected on the object. The method further includes capturing, with a second camera positioned in the particular location, a test image of the object while the diffused is projected on the object. The method further includes comparing speckles in the reference image to the test image. The projector, first camera and second camera are removably provided to and positioned in a site of the object.

Laser speckle photography for surface tampering detection

Geometries of the structures and objects deviate from their idealized models, while not always visible to the naked eye. Embodiments of the present invention reveal and visualize such subtle geometric deviations, which can contain useful, surprising information. In an embodiment of the present invention, a method can include fitting a model of a geometry to an input image, matting a region of the input image according to the model based on a sampling function, generating a deviation function based on the matted region, extrapolating the deviation function to an image wide warping field, and generating an output image by warping the input image according to the warping. In an embodiment of the present invention, Deviation Magnification inputs takes a still image or frame, fits parametric models to objects of interest, and generates an output image exaggerating departures from ideal geometries.

Deviation magnification: revealing departures from ideal geometries

In an embodiment, a method converts two images to a transform representation in a transform domain. For each spatial position, the method examines coefficients representing a neighborhood of the spatial position that is spatially the same across each of the two images. The method calculates a first vector in the transform domain based on first coefficients representing the spatial position, the first vector representing change from a first to second image of the two images describing deformation. The method modifies the first vector to create a second vector in the transform domain representing amplified movement at the spatial position between the first and second images. The method calculates second coefficients based on the second vector of the transform domain. From the second coefficients, the method generates an output image showing motion amplified according to the second vector for each spatial position between the first and second images.

Linear-based Eulerian motion modulation

The method for dynamic video magnification magnifies small motions occurring simultaneously within large motions. The method involves selecting a region of interest from a video for magnification. The region of interest is warped to obtain a stabilized sequence of frames that discounts large motions. Each frame of the stabilized sequence is decomposed to a foreground layer, a background layer, and an alpha matte layer, and each of the foreground and alpha matte layers is magnified. Then a magnified sequence is generated from the magnified layers using matte inversion. Any image holes in the magnified sequence are filled in using texture synthesis. Finally, the magnified sequence is de-warped to the original space-time coordinates.

Method for dynamic video magnification

A method of recovering audio signals and corresponding apparatus according to an embodiment of the present invention using video or other sequence of images enables recovery of sound that causes vibrations of a surface. An embodiment method includes combining representations of local motions of a surface to produce a global motion signal of the surface. The local motions are captured in a series of images of features of the surface, and the global motion signal represents a sound within an environment in which the surface is located. Some embodiments compare representations of local motions of a surface to determine which motions are in-phase or out-of-phase with each other, enabling visualization of surface vibrational modes. Embodiments are passive, as compared to other forms of remote audio recovery that employ active sensing, such as laser microphone systems. Example applications for the embodiments include espionage and surveillance.

Method and apparatus for recovering audio signals from images

A method and corresponding device for identifying operational mode shapes of an object in a video stream includes extracting pixel-wise Eulerian motion signals of an object from an undercomplete representation of frames within a video stream. Pixel-wise Eulerian motion signals are downselected to produce a representative set of Eulerian motion signals of the object. Operational mode shapes of the object are identified based on the representative set. Resonant frequencies can also be identified. Embodiments enable vibrational characteristics of objects to be determined using video in near real time.

Video-based identification of operational mode shapes

Linear-based eulerian motion modulation

Growing community of inventors

Somerville, MA, United States of America

Frederic Durand