Honda Motor Co., Ltd.

The University of North Carolina at Chapel Hill

Eth Zurich

Jongwoo Lim

Ming-Hsuan Yang

David Ross

Takahiro Ohashi

Ruei-Sung Lin

Vivek Pradeep

Rakesh Gupta

Jan-Michael Frahm

Ananth Ranganathan

Marc Pollefeys

Benjamin Laxton

Jeffrey Ho

Manmohan Chandraker

The prevention of vehicle accidents is targeted. A road texture model is created based on a vehicle camera image. An initial vehicle location estimate is determined, and map imagery is obtained based on this location estimate. A refined vehicle location is determined using visual egomotion. In particular, 3D features of the vehicle image and the retrieved map imagery are identified and aligned. A map image is selected based on this alignment, and the location associated with the map image is modified by a displacement between the selected map image and the vehicle image to produce a refined vehicle location. A road boundary model is created based on the road texture model and the refined vehicle location, and a road departure model is created based on the road boundary model and vehicle odometry information. The operator of the vehicle is warned of a road departure based on the road departure model.

Road departure warning system

A system and method are disclosed for online mapping of large-scale environments using a hybrid representation of a metric Euclidean environment map and a topological map. The system includes a scene module, a location recognition module, a local adjustment module and a global adjustment module. The scene flow module is for detecting and tracking video features of the frames of an input video sequence. The scene flow module is also configured to identify multiple keyframes of the input video sequence and add the identified keyframes into an initial environment map of the input video sequence. The location recognition module is for detecting loop closures in the environment map. The local adjustment module enforces local metric properties of the keyframes in the environment map, and the global adjustment module is for optimizing the entire environment map subject to global metric properties of the keyframes in the keyframe pose graph.

Online environment mapping

A system and method are disclosed for estimating camera motion of a visual input scene using points and lines detected in the visual input scene. The system includes a camera server comprising a stereo pair of calibrated cameras, a feature processing module, a trifocal motion estimation module and an optional adjustment module. The stereo pair of the calibrated cameras and its corresponding stereo pair of camera after camera motion form a first and a second trifocal tensor. The feature processing module is configured to detect points and lines in the visual input data comprising a plurality of image frames. The feature processing module is further configured to find point correspondence between detected points and line correspondence between detected lines in different views. The trifocal motion estimation module is configured to estimate the camera motion using the detected points and lines associated with the first and the second trifocal tensor.

Egomotion using assorted features

A system and method are disclosed for estimating camera motion and structure reconstruction of a scene using lines. The system includes a line detection module, a line correspondence module, a temporal line tracking module and structure and motion module. The line detection module is configured to detect lines in visual input data comprising a plurality of image frames. The line correspondence module is configured to find line correspondence between detected lines in the visual input data. The temporal line tracking module is configured to track the detected lines temporally across the plurality of the image frames. The structure and motion module is configured to estimate the camera motion using the detected lines in the visual input data and to reconstruct three-dimensional lines from the estimated camera motion.

Structure and motion with stereo using lines

A system and method are disclosed for calibrating a plurality of projectors for three-dimensional scene reconstruction. The system includes a plurality of projectors and at least one camera, a camera-projector calibration module and a projector-projector calibration module. The camera-projector calibration module is configured to calibrate a first projector with the camera and generate a first camera-projector calibration data using camera-projector duality. The camera-projector calibration module is also configured to calibrate a second projector with the camera and generate a second camera-projector calibration data. The projector-projector calibration module is configured to calibrate the first and the second projector using the first and the second camera-projector calibration data.

Camera-projector duality: multi-projector 3D reconstruction

A system (and a method) are disclosed for recognizing and representing activities in a video sequence. The system includes an activity dynamic Bayesian network (ADBN), an object/action dictionary, an activity inference engine and a state output unit. The activity dynamic Bayesian network encodes the prior information of a selected activity domain. The prior information of the selected activity domain describes the ordering, temporal constraints and contextual cues among the expected actions. The object/action dictionary detects activities in each frame of the input video stream, represents the activities hierarchically, and generates an estimated observation probability for each detected action. The activity inference engine estimates a likely activity state for each frame based on the evidence provided by the object/action dictionary and the ADBN. The state output unit outputs the likely activity state generated by the activity inference engine.

Leveraging temporal, contextual and ordering constraints for recognizing complex activities in video

A system and method are disclosed for generating an optimized projection pattern and for using the optimized projection pattern for depth reconstruction. The system includes a De Bruijn graph generation module, a non-recurring De Bruijn sequence generation module and projection pattern generation module. The De Bruijn graph generation module is configured to generate a classical De Bruijn graph. The non-recurring De Bruijn sequence generation module is configured to generate a non-recurring De Bruijn sequence by eliminating nodes with recurring alphabets from the classical De Bruijn sequence and calculating a Hamiltonian cycle of the modified De Bruijn graph. The projection pattern generation module is configured to generate the optimized projection pattern form the non-recurring De Bruijn sequence. The system further comprises a projector to project the non-recurring De Bruijn sequence to a plurality of images and a depth reconstruction module to reconstruct depth images from the plurality of the images.

Optimized projection pattern for long-range depth sensing

A system and a method model the motion of a non-rigid object using a thin plate spline (TPS) transform. A first image of a video sequence is received, and a region of interest, referred to as a template, is chosen manually or automatically. A set of arbitrarily-chosen fixed reference points is positioned on the template. A target image of the video sequence is chosen for motion estimation relative to the template. A set of pixels in the target image corresponding to the pixels of the template is determined, and this set of pixels is back-warped to match the template using a thin-plate-spline-based technique. The error between the template and the back-warped image is determined and iteratively minimized using a gradient descent technique. The TPS parameters can then be used to estimate the relative motion between the template and the corresponding region of the target image. According to one embodiment, a stiff-to-flexible approach mitigates instability that can arise when reference points lie in textureless regions, or when the initial TPS parameters are not close to the desired ones. The value of a regularization parameter is varied from a larger to a smaller value, varying the nature of the warp from stiff to flexible, so as to progressively emphasize local non-rigid deformations.

Direct method for modeling non-rigid motion with thin plate spline transformation

A system and a method are disclosed for adaptive probabilistic tracking of an object within a motion video. The method utilizes a time-varying Eigenbasis and dynamic, observation and inference models. The Eigenbasis serves as a model of the target object. The dynamic model represents the motion of the object and defines possible locations of the target based upon previous locations. The observation model provides a measure of the distance of an observation of the object relative to the current Eigenbasis. The inference model predicts the most likely location of the object based upon past and present observations. The method is effective with or without training samples. A computer-based system provides a means for implementing the method. The effectiveness of the system and method are demonstrated through simulation.

Adaptive probabilistic visual tracking with incremental subspace update

The face detection system and method attempts classification of a test image before performing all of the kernel evaluations. Many subimages are not faces and should be relatively easy to identify as such. Thus, the SVM classifier try to discard non-face images using as few kernel evaluations as possible using a cascade SVM classification. In the first stage, a score is computed for the first two support vectors, and the score is compared to a threshold. If the score is below the threshold value, the subimage is classified as not a face. If the score is above the threshold value, the cascade SVM classification function continues to apply more complicated decision rules, each time doubling the number of kernel evaluations, classifying the image as a non-face (and thus terminating the process) as soon as the test image fails to satisfy one of the decision rules. Finally, if the subimage has satisfied all intermediary decision rules, and has now reached the point at which all support vectors must be considered, the original decision function is applied. Satisfying this final rule, and all intermediary rules, is the only way for a test image to garner a positive (face) classification.

Face recognition system

A system and a method are disclosed for an adaptive discriminative generative model with a probabilistic interpretation. As applied to visual tracking, the discriminative generative model separates the target object from the background more accurately and efficiently than conventional methods. A computationally efficient algorithm constantly updates the discriminative model over time. The discriminative generative model adapts to accommodate dynamic appearance variations of the target and background. Experiments show that the discriminative generative model effectively tracks target objects undergoing large pose and lighting changes.

Adaptive discriminative generative model and application to visual tracking

A system and a method are disclosed for clustering images of objects seen from different viewpoints. That is, given an unlabelled set of images of n objects, an unsupervised algorithm groups the images into N disjoint subsets such that each subset only contains images of a single object. The clustering method makes use of a broad geometric framework that exploits the interplay between the geometry of appearance manifolds and the symmetry of the 2D affine group.

Image clustering with metric, local linear structure, and affine symmetry

The advantage of the present invention is to appropriately detect the object. The object detection apparatus in the present invention has a plurality of cameras to determine the distance to the objects, a distance determination unit to determine the distance therein, a histogram generation unit to specify the frequency of the pixels against the distances to the pixels, an object distance determination unit that determines the most likely distance, a probability mapping unit that provides the probabilities of the pixels based on the difference of the distance, a kernel detection unit that determines a kernel region as a group of the pixels, a periphery detection unit that determines a peripheral region as a group of the pixels, selected from the pixels being close to the kernel region and an object specifying unit that specifies the object region where the object is present with a predetermined probability.

Growing community of inventors

Sunnyvale, CA, United States of America

Jongwoo Lim