Microsoft Technology Licensing, LLC

Vladimir Kolmogorov

Andrew Blake

Carsten Curt Eckard Rother

Antonio Criminisi

Geoffrey Cross

Toby Leonard Sharp

Carsten Rother

Victor Sergeyevich Lempitsky

Sanjiv Kumar

Methods of image segmentation using reduced foreground training data are described. In an embodiment, the foreground and background training data for use in segmentation of an image is determined by optimization of a modified energy function. The modified energy function is the energy function used in image segmentation with an additional term comprising a scalar value. The optimization is performed for different values of the scalar to produce multiple initial segmentations and one of these segmentations is selected based on pre-defined criteria. The training data is then used in segmenting the image. In other embodiments further methods are described: one places an ellipse inside the user-defined bounding box to define the background training data and another uses a comparison of properties of neighboring image elements, where one is outside the user-defined bounding box, to reduce the foreground training data.

Image segmentation using reduced foreground training data

Methods of updating image segmentation following user input are described. In an embodiment, the properties used in computing the different portions of the image are updated as a result of one or more user inputs. Image elements which have been identified by a user input are given more weight when updating the properties than other image elements which have already been assigned to a particular portion of the image. In another embodiment, an updated segmentation is post-processed such that only regions which are connected to an appropriate user input are updated.

Updating image segmentation following user input

Segmentation of foreground from background layers in an image may be provided by a segmentation process which may be based on one or more factors including motion, color, contrast, and the like. Color, motion, and optionally contrast information may be probabilistically fused to infer foreground and/or background layers accurately and efficiently. A likelihood of motion vs. non-motion may be automatically learned from training data and then fused with a contrast-sensitive color model. Segmentation may then be solved efficiently by an optimization algorithm such as a graph cut. Motion events in image sequences may be detected without explicit velocity computation.

Image segmentation of foreground from background layers

Computer vision applications often require each pixel within an image to be assigned one of a set of labels. A method of improving the labels assigned to pixels is described which uses the quadratic pseudoboolean optimization (QPBO) algorithm. Starting with a partially labeled solution, an unlabeled pixel is assigned a value from a fully labeled reference solution and the energy of the partially labeled solution plus this additional pixel is calculated. The calculated energy is then used to generate a revised partially labeled solution using QPBO.

Optimizing pixel labels for computer vision applications

Previously, Poisson blending has been used for image blending including cloning an object onto a target background and blending pairs of source images together. Such Poisson blending works well in many situations. However, whilst this method is always workable, we have found that discolorations sometimes occur. We realized that these discolorations occur when the gradient of the source image is preserved too insistently, at the expense of preserving object and background color. In some situations object outlines become smeared or blurred. We develop a color preservation term and a fragility measure to address these problems. This gives a user additional control to obtain smooth compositions and reduce discoloration artifacts.

Image blending

Real-time segmentation of foreground from background layers in binocular video sequences may be provided by a segmentation process which may be based on one or more factors including likelihoods for stereo-matching, color, and optionally contrast, which may be fused to infer foreground and/or background layers accurately and efficiently. In one example, the stereo image may be segmented into foreground, background, and/or occluded regions using stereo disparities. The stereo-match likelihood may be fused with a contrast sensitive color model that is initialized or learned from training data. Segmentation may then be solved by an optimization algorithm such as dynamic programming or graph cut. In a second example, the stereo-match likelihood may be marginalized over foreground and background hypotheses, and fused with a contrast-sensitive color model that is initialized or learned from training data. Segmentation may then be solved by an optimization algorithm such as a binary graph cut.

Stereo image segmentation

Segmentation of foreground from background layers in an image may be provided by a segmentation process which may be based on one or more factors including motion, color, contrast, and the like. Color, motion, and optionally contrast information may be probabilistically fused to infer foreground and/or background layers accurately and efficiently. A likelihood of motion vs. non-motion may be automatically learned from training data and then fused with a contrast-sensitive color model. Segmentation may then be solved efficiently by an optimization algorithm such as a graph cut.

An output image formed from at least a portion of one or more input images may be automatically synthesized as a tapestry image. To determine which portion or region of each input image will be used in the image tapestry, the regions of each image may be labeled by one of a plurality of labels. The multi-class labeling problem of creating the tapestry may be resolved such that each region in the tapestry is constructed from one or more salient input image regions that are selected and placed such that neighboring blocks in the tapestry satisfy spatial compatibility. This solution may be formulated using a Markov Random Field and the resulting tapestry energy function may be optimized in any suitable manner. To optimize the tapestry energy function, an expansion move algorithm for energy functions may be generated to apply to non-metric hard and/or soft constraints.

Image tapestry

Techniques are disclosed to provide more efficient and improved border matting for extracted foreground images, e.g., without requiring excessive user interaction. Border matting techniques described herein generate relatively continuous transparency (or alpha values) along the boundary of the extracted object (e.g., limiting color bleeding and/or artifacts).

Growing community of inventors

Ipswich, United Kingdom

Vladimir Kolmogorov