Microsoft Technology Licensing, LLC

Other

Carsten Rother

Antonio Criminisi

Andrew Blake

Stephen Edward Hodges

John Michael Winn

Lyndsay Williams

Kenneth Robert Wood

James Srinivasan

Victor Lempitsky

Pushmeet Kohli

Vladimir Kolmogorov

Victor Sergeyevich Lempitsky

Andrew William Fitzgibbon

Stuart Taylor

Anitha Kannan

John Herbert Chiloyan

Jamie Shotton

Youssef Hamadi

Lucas Bordeaux

Alexander Rav-Acha

Gavin Smyth

Mitch Goldberg

Varun Gulshan

Amit Shesh

William H De Boer

Systems and methods for editing digital images using information about objects in those images are described. For example, the information about objects comprises depth ordering information and/or information about the class each object is a member of. Examples of classes include sky, building, aeroplane, grass and person. This object-level information is used to provide new and/or improved editing functions such as cut and paste, filling-in image regions using tiles or patchworks, digital tapestry, alpha matte generation, super resolution, auto cropping, auto color balance, object selection, depth of field manipulation, and object replacement. In addition improvements to user interfaces for image editing systems are described which use object-level information.

Object-level image editing

A method and apparatus for processing video is disclosed. In an embodiment, image features of an object within a frame of video footage are identified and the movement of each of these features is tracked throughout the video footage to determine its trajectory (track). The tracks are analyzed, the maximum separation of the tracks is determined and used to determine a texture map, which is in turn interpolated to provide an unwrap mosaic for the object. The process may be iterated to provide an improved mosaic. Effects or artwork can be overlaid on this mosaic and the edited mosaic can be warped via the mapping, and combined with layers of the original footage. The effect or artwork may move with the object's surface.

Video processing

An image capture device is described which is operable in any one of a number of capture modes. The device comprises a camera, a memory and a processor. The memory stores a plurality of sets of capture triggers, with each set of capture triggers being associated with one of the plurality of capture modes. The processor selects one of the plurality of capture modes, such that the device is operable in the selected capture mode. In the selected capture mode, an image is captured automatically when a capture trigger within the associated set of capture triggers is satisfied.

Automatic capture modes

A method of transferring images from a first device to a second device and computer program code for performing this method is described. A connection characteristic for a connection between the first & second devices is determined and at least one image is selected from a plurality of images on the first device for transfer dependent upon both the connection characteristic and image selection criteria. The selected image(s) are then transferred over the connection from the first device to the second device.

Transferring of digital images

A method of labeling pixels in an image is described where the pixel label is selected from a set of three or more labels. The pixel labeling problem is reduced to a sequence of binary optimizations by representing the label value for each pixel as a binary word and then optimizing the value of each bit within the word, starting with the most significant bit. Data which has been learned from one or more training images is used in the optimization to provide information about the less significant bits within the word.

Optimization of multi-label problems in computer vision

Image segmentation using star-convexity constraints is described. In an example, user input specifies positions of one or more star centers in a foreground to be segmented from a background of an image. In embodiments, an energy function is used to express the problem of segmenting the image and that energy function incorporates a star-convexity constraint which limits the number of possible solutions. For example, the star-convexity constraint may be that, for any point p inside the foreground, all points on a shortest path (which may be geodesic or Euclidean) between the nearest star center and p also lie inside the foreground. In some examples continuous star centers such as lines are used. In embodiments a user may iteratively edit the star centers by adding brush strokes to the image in order to progressively change the star-convexity constraints and obtain an accurate segmentation.

Image segmentation using star-convexity constraints

Image editing which is consistent with geometry of a scene depicted in the image is described. In an embodiment a graphical user interface (GUI) is provided to enable a user to simply and quickly specify four corners of a rectangular frame drawn onto a source image using the GUI. In embodiments, the four corners are used to compute parameters of a virtual camera assumed to capture the image of the drawn frame. Embodiments of an image processing system are described which use the virtual camera parameters to control editing of the source image in ways consistent with the 3D geometry of the scene depicted in that image. In some embodiments out of bounds images are formed and/or realistic-looking shadows are synthesized. In examples, users are able to edit images and the virtual camera parameters are dynamically recomputed and used to update the edited image.

Image editing consistent with scene geometry

An image processing system is described which automatically labels image elements of a digital image. In an embodiment an energy function describing the quality of possible labelings of an image is globally optimized to find an output labeled image. In the embodiment, the energy function comprises terms that depend on at least one non-local parameter. For example, the non-local parameter describes characteristics of image elements having the same label. In an embodiment the global optimization is achieved in a practical, efficient manner by using a tree structure to represent candidate values of the non-local parameter and by using a branch and bound process. In some embodiments, the branch and bound process comprises evaluating a lower bound of the energy function by using a min-cut process. For example, the min-cut process enables the lower bound to be evaluated efficiently using a graphical data structure to represent the lower bound.

Labeling image elements

A method of displaying digital images is described in which a display length indicator is received and digital images are accessed. A set of digital images are selected from the accessed digital images in accordance with the display length indicator and displayed in a predetermined order. The method may be performed by a computer program, which may be embodied on a computer readable medium.

Playback of digital images

Multi-scale processing may be used to reduce the memory and computational requirements of optimization algorithms for image labeling, for example, for object segmentation, 3D reconstruction, stereo correspondence, optical flow and other applications. For example, in order to label a large image (or 3D volume) a multi-scale process first solves the problem at a low resolution, obtaining a coarse labeling of an original high resolution problem. This labeling is refined by solving another optimization on a subset of the image elements. In examples, an energy function for a coarse level version of an input image is formed directly from an energy function of the input image. In examples, the subset of image elements may be selected using a measure of confidence in the labeling.

Image labeling using multi-scale processing

Computer vision applications often require each pixel within an image to be assigned one of a set of labels. A method of improving the labels assigned to pixels is described which uses the quadratic pseudoboolean optimization (QPBO) algorithm. Starting with a partially labeled solution, an unlabeled pixel is assigned a value from a fully labeled reference solution and the energy of the partially labeled solution plus this additional pixel is calculated. The calculated energy is then used to generate a revised partially labeled solution using QPBO.

Optimizing pixel labels for computer vision applications

Previously, Poisson blending has been used for image blending including cloning an object onto a target background and blending pairs of source images together. Such Poisson blending works well in many situations. However, whilst this method is always workable, we have found that discolorations sometimes occur. We realized that these discolorations occur when the gradient of the source image is preserved too insistently, at the expense of preserving object and background color. In some situations object outlines become smeared or blurred. We develop a color preservation term and a fragility measure to address these problems. This gives a user additional control to obtain smooth compositions and reduce discoloration artifacts.

Image blending

Given an image of structured and/or unstructured objects we automatically partition it into semantically meaningful areas each labeled with a specific object class. We use a novel type of feature which we refer to as a shape filter. Shape filters enable us to capture some or all of shape, texture and appearance context information. A shape filter comprises one or more regions of arbitrary shape, size and position within a bounding area of an image, paired with a specified texton. A texton comprises information describing the texture of a patch of surface of an object. In a training process we select a sub-set of possible shape filters and incorporate those into a conditional random field model of object classes. That model is then used for object detection and recognition.

Object recognition using textons and shape filters

Many problems in the fields of image processing and computer vision relate to creating good representations of information in images of objects in scenes. We provide a system for learning repeated-structure elements from one or more input images. The repeated-structure elements are patches that may be single pixels or coherent groups of pixels of varying shape, size and appearance (where those shapes and sizes are not pre-specified). Input images are mapped to a single output image using offset maps to specify the mapping. A joint probability distribution on the offset maps, output image and input images is specified and an unsupervised learning process is used to learn the offset maps and output image. The learnt output image comprises repeated-structure elements. This shape and appearance information captured in the learnt repeated-structure elements may be used for object recognition and many other tasks.

Growing community of inventors

Cambridge, United Kingdom

Carsten Rother