Mitsubishi Electric Research Laboratories, Inc.

Mitsubishi Electric Research Labs.

Mitsubishi Electric Information Technology Center America, Inc (ita)

Mitsubishi Denki Kabushiki Kaisha

Paul A Beardsley

Ramesh Raskar

Jeroen Van Baar

Wojciech Matusik

Clifton L Forlines

Hanspeter Pfister

Leonard McMillan, Jr

Joseph W Marks

Paul H Dietz

Chia Shen

Baback Moghaddam

Dirk Brinkman

David B Anderson

Ryan Scott Bardsley

Vaibhav Vaish

Neal B Lesh

Wai Kit Addy Ngan

Aseem Agarwala

Moritz Baecher

A camera acquires a set of coded images and a set of flash images of a semi-specular object. The coded images are acquired while scanning the object with a laser beam pattern, and the flash images are acquired while illuminating the object with a set of light sources at different locations near the camera, there being one flash image for each light source. 3D coordinates of points on the surface of the object are determined from the set of coded images, and 2D silhouettes of the object are determined from shadows cast in the set of flash images. Surface normals are obtained for the 3D points from photometric stereo on the set of flash images. The 3D coordinates, 2D silhouettes and surface normals are compared with a known 3D model of the object to determine the pose of the object.

Method and system for determining poses of semi-specular objects

Surface normals and other 3D shape descriptors are determined for a specular or hybrid specular-diffuse object. A camera records an image of a smoothly spatially-varying pattern being reflected in the surface of the object, with the pattern placed at an initial position. The camera then records multiple images of the pattern undergoing a sequence of subsequent displacements to a final position distinct from the initial position. For a pixel in the images, the pattern displacement that corresponds to the minimum difference between the pixel value in the initial image and any of the final images is determined. The incident ray that strikes the surface of the object at the point being imaged by the pixel is then determined using the determined pattern displacement. The surface normal at that same surface point is then determined using the determined incident ray. Other 3D shape descriptors are determined from the surface normals of the object, such as principal curvatures, or the principal axes of a parametric surface, or the 3D shape, or the object pose.

Method and system for sensing 3D shapes of objects with specular and hybrid specular-diffuse surfaces

A method generates input text with a pointing device at a distance from a writing surface. Motion information of a mobile coordinate frame of the pointing device is sensed relative to a stationary coordinate frame of the writing surface using a motion sensor configured to track motion of the pointing device. The input text is then determined from the motion information. A projection of a track of the motion of the pointing device on the writing surface can also be determined from the motion information, and the track can be displayed on the writing surface as the input text.

Motion-based text input

A method displays an output image on an object. A set of unique markers are fixed to the object at predetermined locations. An input image of the object and the markers is acquired with a camera in a fixed physical relationship to a projector. A pose of the projector with respect to the markers is determined from the image. Then, one or more output images related to the object can be projected onto the object, at predetermine locations, according to the pose of the projector and the unique markers.

Context aware projector

A method determines a shape of an object in from an image. A visual hull larger than the object is constructed such that the object can be entirely contained within the visual hull. An image is acquired of the object by a camera. The visual hull is projected onto the image according to a position and pose of the camera. Seed pixels in the image are selected. The seed pixels are located on the projected visual hull, and a region is grown around each seed pixel. Each region includes neighboring pixels resembling the seed pixel. Then, the visual hull is reduced in size according to the grown regions to determine a two-dimensional shape of the object. When multiple images are used from different points of view, the three-dimensional shape can be determined.

Object segmentation from images acquired by handheld cameras

A position and orientation of a projector are determined from projected images. First, a homography between a camera and a planar surface, where the relationship between the camera and planar surface is fixed, is determined. Next, a known pattern is projected on the planar surface with a projector having known intrinsic parameters and an unknown position and orientation. An image of the known pattern is acquired with the camera, and a position and orientation of the projector is determined from the image using the homography and the intrinsic parameters.

Position and orientation sensing with a projector

A method links edges in stereo images into chains which correspond to physical contours in a scene. First, edges are detected in the stereo images. Edges between the stereo images are matched, such that each edge has associated stereo matches. The edges are linked into chains, such that each chained edge has a single associated stereo match, and neighboring edges in a chain have matches with a substantial similar stereo disparity.

Edge chaining using smoothly-varying stereo disparity

A wireless location and identification system includes a controller and a tag. The controller has a projector configured to generate a unique temporal sequence of light intensities for each pixel in an output image of the projector and a wireless transceiver configured to emit an output signal and receive an input signal. The tag includes a photo-sensor configured to sense the unique temporal sequence of light intensities of a particular pixel, a transducer configured to receive the output signal from the transceiver, and a means for modulating, according to the unique temporal sequence of light intensities, the input signal for the transceiver in response to receiving the output signal and sensing the unique temporal sequence of light intensities of the particular pixel to indicate a location of the tag to the controller.

Interactive wireless tag location and identification system

A rigidly-coupled projector-camera system is augmented with laser pointers. The projector-camera system and lasers can be used to determine homography between the projector and a display surface, so that a projection of any desired geometry can be made on the display surface.

Projector-camera system with laser pointers

A method classifies an object in a scene as either an ambulatory person or a wheelchair user. Images of the scene are acquired with a set of stereo cameras. Moving objects in the scene are segmented from the background using detected edges. A 3D reconstruction of the object is generated from the edges, and tests are performed on height and 3D shape of the 3D reconstruction to classify the object as the ambulatory person or the wheelchair user.

Wheelchair detection using stereo vision

A method detects edges of an object in a scene by first acquiring a static image of the scene when the scene is static, and a live image of the scene including the object. A reference image is constructed from the static image. Then, image gradients of the reference image are compared with image gradients of the live image to identify edges of the object in the live image.

Edge detection based on background change

In a method and system for calibrating a multiple cameras, a calibration object is placed on a turntable while a set of images is acquired with each camera while rotating the turntable. Vertices of a calibration pattern are located in each set of images, and intrinsic parameters for each camera are determined from the vertices. An orientation of a rotation axis of the turntable relative to a position of each camera is estimated from the intrinsic parameters of each camera. A dependency of each orientation on a global geometry of the calibration pattern is minimized. Then, one camera is selected as a fixed camera, all other cameras are designated as floating cameras. A translation vector between the fixed camera and each floating camera is determined, and the translation vectors are used to place the cameras in a common coordinate frame according to the translating vectors.

Calibration of multiple cameras for a turntable-based 3D scanner

A method models a three-dimensional object by first acquiring alpha mattes of the object for multiple viewpoints. The alpha mattes are then projected onto a surface hull completely enclosing the object to construct an opacity hull storing opacity values of the surface of the object. The object is illuminated for various lighting conditions while images are acquired. The images are projected onto the opacity hull to render the object under arbitrary lighting conditions for arbitrary viewpoints.

Image-based 3D modeling rendering system

A system projects multiple output images on a display surface. The system includes multiple projectors. Each projector includes a processing unit. The processing unit includes a microprocessor, a memory and an I/O interface connected by busses. A projector sub-system coupled to the processing unit is for displaying output images on the display surface. A camera sub-system coupled to the processing unit is for acquiring input images reflecting a geometry of the display surface. The camera sub-system is in a fixed physical relationship to the projector sub-system. A network sub-system coupled to the processing unit is for communicating messages between the projectors. Internal sensors coupled to the processing unit are for determining an orientation of the projector sub-system and the camera sub-system with respect to the display surface.

Self-configurable ad-hoc projector cluster

An apparatus for projecting an output image on a display surface includes a processing unit. The processing unit includes a microprocessor, a memory and an I/O interface connected by buses. A projector sub-system coupled to the processing unit is for display output images on the display surface. A camera sub-system couple to the processing unit is for acquiring input images reflecting a geometry of the display surface. The camera sub-system is in a fixed physical relationship to the projector sub-system. Internal sensors coupled to the processing unit are for determining an orientation of the projector sub-system and the camera sub-system with respect to the display surface.

Growing community of inventors

Boston, MA, United States of America

Paul A Beardsley