Compaq Computer Corporation, Inc.

Hewlett-Packard Development Company, L.p.

Digital Equipment Corporation

James Matthew Rehg

Tat-Jen Cham

Daniel Dale Morris

Vladimir Pavlovi&cacute;

Sing Bing Kang

Vladimir Pavlović

Robert H Halstead, Jr

Rishiyur S Nikhil

Leonidas Kontothanassis

Keith Waters

Umakishore Ramachandran

Christopher F Joerg

Maria Loughlin

Henry A Rowley

A target in a sequence of measurements is tracked by modeling the target with a switching linear dynamic system (SLDS) having a plurality of dynamic models. Each dynamic model is associated with a switching state such that a model is selected when its associated switching state is true. A set of continuous state estimates is determined for a given measurement, and for each possible switching state. A state transition record is then determined by determining and recording, for a given measurement and for each possible switching state, an optimal previous switching state, based on the measurement sequence, where the optimal previous switching state optimizes a transition probability based on the set of continuous state estimates. A measurement model of the target is fitted to the measurement sequence. The measurement model is the description of the influence of the state on the measurement. It couples what is observed to the estimated target. Finally, a trajectory of the target is estimated from the measurement model fitting, the state transition record and parameters of the SLDS, where the estimated trajectory is a sequence of continuous state estimates of the target which correspond to the measurement sequence. The set of continuous state estimates is preferably obtained through Viterbi prediction. The optimal previous switching state can be an optimal prior switching state, or can be an optimal posterior switching state.

Method for visual tracking using switching linear dynamic systems models

A method for synthesizing a sequence includes defining a switching linear dynamic system (SLDS) with a plurality of dynamic systems. In a Viterbi-based method, a state transition record for a training sequence is determined. The corresponding sequence of switching states is determined by backtracking through the state transition record. Parameters of dynamic models are learned in response to the determined sequence of switching states, and a new data sequence is synthesized, based on the dynamic models whose parameters have been learned. In a variational-based method, the switching state at a particular instance is determined by a switching model. The dynamic models are decoupled from the switching model, and parameters of the decoupled dynamic model are determined responsive to a switching state probability estimate. Similar methods are used to interpolate from an input sequence.

Method for motion synthesis and interpolation using switching linear dynamic system models

Portions of an input measurement sequence are classified into a plurality of regimes by associating each of a plurality of dynamic models with one a switching state such that a model is selected when its associated switching state is true. In a Viterbi-based method, a state transition record is determined, based on the input sequence. A switching state sequence is determined by backtracking through the state transition record. Finally, portions of the input sequence are classified into different regimes, responsive to the switching state sequence. In a variational-based method, the switching state at a particular instance is also determined by a switching model. The dynamic model is then decoupled from the switching model. Parameters of the decoupled dynamic model are determined responsive to a switching state probability estimate. A state of the decoupled dynamic model corresponding to a measurement at the particular instance is estimated, responsive to the input sequence. Parameters of the decoupled switching model are then determined responsive to the dynamic state estimate. A probability is estimated for each possible switching state of the decoupled switching model. A switching state sequence is determined based on the estimated switching state probabilities. Finally, portions of the input sequence are classified into different regimes, responsive to the determined switching state sequence.

Method for motion classification using switching linear dynamic systems models

An object model has a plurality of features and is described by a model state. An unregistered feature of the object model, and an available frame from a sequence of images are selected to minimize a cost function of a subsequent search for a match of the selected model feature to the image in the selected frame. Upon a match, the feature is registered in that frame. The model state is then updated for each available frame. The steps of selecting, searching and updating are repeated. A video storage module may contain only one frame corresponding to a single time instance, in which case the framework used is based on integrated sequential feature selection. Alternatively, the video store may contain the entire video sequence, in which case feature selection is performed across all video frames for maximum tracking efficiency. Finally, the video store may contain a small number of previous frames plus the current frame, in which case feature selection spans only a subset of the entire video sequence for each feature matching cycle.

Method for efficiently tracking object models in video sequences via dynamic ordering of features

Method for motion classification using switching linear dynamic system models

Method for visual tracking using switching linear dynamic system models

An object model, having a plurality of features and described by a model state, is registered in an image. Unregistered features of the object model are dynamically selected such that the cost function of each feature search is minimized. A search is performed for a match of the selected model feature to the image, or to features within the image, to register the feature, and the model state is updated accordingly. These steps are repeated until all features have been registered. The search is performed in a region of high probability of a match. The cost function for a feature is based on the feature's basin of attraction, and in particular can be based on the complexity of the search process at each basin of attraction. A search region is based on a projected state probability distribution. In particular, the cost function is based on the &ldquo;matching ambiguity,&rdquo; or the number of search operations required to find a true match with some specified minimum probability. For feature-to-feature matching, the number of search operations is preferably the number of target features located within each search region. For feature-to-image matching, the matching ambiguity is computed, for each search region, by dividing the region into minimally-overlapping volumes which have the same size and shape as a basin of attraction associated with the feature, and then counting the number of volumes required to cover the regions. The model state is updated according to a propagated state probability distribution. Preferably, the propagation of the probability distribution is based on successive registered features.

Method for efficiently registering object models in images via dynamic ordering of features

A plurality of object models, where each object model comprises a plurality of features and is described by a model state, are registered in at least one image a subset of the object models is selected. Different object models have different sets of features, which may or may not overlap. A feature of each selected object model is registered in one of the images, and the model state for each selected object model is updated accordingly. The model states of some or all of the object models are then updated according to a set of constraints. These steps are repeated until one or more object models are registered. At the beginning of each registration cycle, a cost function of a subsequent search is determined for each unregistered feature of each object model. An unregistered feature of each object model is then selected such that the cost function is minimized. Object models to which the selected features belong are then selected, and each selected object model's selected feature is registered by matching it to an image. The selected unregistered features are ranked according to some criterion, such as the number of operations needed to search for a feature, i.e., the matching ambiguity. Object models are then selected according to the ranking. Preferably, a predetermined number of object models is selected each cycle.

Method for object registration via selection of models with dynamically ordered features

From a set of possible switching states and responsive to a sequence of measurements, a corresponding sequence of switching states is determined for a system having a plurality of dynamic models, associates each model with a switching state such that a model is selected when its associated switching state is true. A state transition record is determined, based on the measurement sequence. The sequence of switching states is determined by backtracking through the state transition record. Alternatively, the switching state model is decoupled from the dynamic system model. The decoupled switching state model is transformed into a hidden Markov model (HMM) switching state model, while the decoupled dynamic system model is transformed into a time-varying dynamic system model. A solution to the dynamic system model is estimated using a Kalman filter. Next, variational parameters of the HMM switching state model are determined based on the estimated-solution, where the variational parameters measure an agreement of each model from the plurality of dynamic models with the solution. A sequence of switching states for the HMM switching state model is then determined based on the variational parameters of the HMM switching state model. finally, variational parameters of the dynamic system model are determined based on the determined sequence of switching states, such that the variational parameters are proportional to a combination of model parameters form the plurality of dynamic models weighted by the probability of the switching states.

Method for learning switching linear dynamic system models from data

The invention recognizes that a probability density function for fitting a model to a complex set of data often has multiple modes, each mode representing a reasonably probable state of the model when compared with the data. Particularly, sequential data such as are collected from detection of moving objects in three dimensional space are placed into data frames. Also, a single frame of data may require analysis by a sequence of analysis operations. Computation of the probability density function of the model state involves two main stages: (1) state prediction, in which the prior probability distribution is generated from information known prior to the availability of the data, and (2) state update, in which the posterior probability distribution is formed by updating the prior distribution with information obtained from observing the data. In particular this information obtained purely from data observations can also be expressed as a probability density function, known as the likelihood function. The likelihood function is a multimodal (multiple peaks) function when a single data frame leads to multiple distinct measurements from which the correct measurement associated with the model cannot be distinguished. The invention analyzes a multimodal likelihood function by numerically searching the likelihood function for peaks. The numerical search proceeds by randomly sampling from the prior distribution to select a number of seed points in state-space, and then numerically finding the maxima of the likelihood function starting from each seed point. Furthermore, kernel functions are fitted to these peaks to represent the likelihood function as an analytic function. The resulting posterior distribution is also multimodal and represented using a set of kernel functions. It is computed by combining the prior distribution and the likelihood function using Bayes Rule. The peaks in the posterior distribution are also referred to as 'hypotheses', as they are hypotheses for the states of the model which best explain both the data and the prior knowledge.

Sample refinement method of multiple mode probability density estimation

The invention recognizes that a probability density function for fitting a model to a complex set of data often has multiple modes, each mode representing a reasonably probable state of the model when compared with the data. Particularly, sequential data such as are collected from detection of moving objects in three dimensional space are placed into data frames. Computation of the probability density function of the model state involves two main stages: (1) state prediction, in which the prior probability distribution is generated from information known prior to the availability of the data, and (2) state update, in which the posterior probability distribution is formed by updating the prior distribution with information obtained from observing the data. In particular this information obtained purely from data observations can also be expressed as a probability density function, known as the likelihood function. The likelihood function is a multimodal (multiple peaks) function when a single data frame leads to multiple distinct measurements from which the correct measurement associated with the model cannot be distinguished. The invention analyzes a multimodal likelihood function by numerically searching the likelihood function for peaks. The numerical search proceeds by randomly sampling from the prior distribution to select a number of seed points in state-space, and then numerically finding the maxima of the likelihood function starting from each seed point. Furthermore, kernel functions are fitted to these peaks to represent the likelihood function as an analytic function. The resulting posterior distribution is also multimodal and represented using a set of kernel functions. It is computed by combining the prior distribution and the likelihood function using Bayes Rule. The peaks in the posterior distribution are also referred to as 'hypotheses', as they are hypotheses for the states of the model which best explain both the data and the prior knowledge.

Multiple mode probability density estimation with application to multiple hypothesis tracking

In a computerized method, a moving articulated figure is tracked in a sequence of 2-D images measured by a monocular camera. The images are individually registered with each other using a 2-D scaled prismatic model of the figure. The 2-D model includes a plurality of links connected by revolute joints to form is a branched, linear-chain of connected links. The registering produces a state trajectory for the figure in the sequence of images. During a reconstructing step, a 3-D model is fitted to the state trajectory to estimate kinematic parameters, and the estimated kinematic parameters are refined using an expectation maximization technique.

Method for tracking the motion of a 3-D figure

A computerized method and related computer system synthesize video from a plurality of sources of image data. The sources include a variety of image data types such a collection of image stills, a sequence of video frames, and 3-D models of objects. Each source provides image data associated with an object. One source provides image data associated with a first object, and a second source provides image data associated with a second object. The image data of the first and second objects are combined to generate composite images of the first and second objects. From the composite images, an output image of the first and second objects as viewed from an arbitrary viewpoint is generated. Gaps of pixels with unspecified pixel values may appear in the output image. Accordingly, a pixel value for each of these &ldquo;missing pixels&rdquo; is obtained by using an epipolar search process to determine which one of the sources of image data should provide the pixel value for that missing pixel.

Multi-layer image-based rendering for video synthesis

An active public user interface in a computerized kiosk senses humans visually using movement and color to detect changes in the environment indicating the presence of people. Interaction spaces are defined and the system records an initial model of its environment which is updated over time to reflect the addition or subtraction of inanimate objects and to compensate for lighting changes. The system develops models of the moving objects and is thereby able to track people as they move about the interaction spaces. A stereo camera system further enhances the system's ability to sense location and movement. The kiosk presents audio and visual feedback in response to what it &ldquo;sees.&rdquo;

Method and apparatus for visual sensing of humans for active public interfaces

A computerized method compresses an input sequence of 2-D images captured by a monocular camera. The input sequence of images includes a moving articulated 3-D figure. A pose of the figure in each image is registered using a 2-D scaled prismatic model to produce a state trajectory for the figure in each image of the input sequence. The state trajectory of the figure in the images is reconstructed to determine kinematic parameters of a 3-D kinematic model that best fit the state trajectories, The 3-D kinematic model represents the 3-D movement of the figure in the input sequence of 2-D images. Static information of a first image of the input sequence is encoded. For subsequent images of the input sequence, a time series of state vectors containing the kinematic parameters of the moving figure are encoded. The static information of the first image and the time series of state vectors can be used to generate an output sequence of images including the moving articulated 3-D figure on a viewing device.

Growing community of inventors

Arlington, MA, United States of America

James Matthew Rehg