Adobe, Inc.

Mitsubishi Electric Research Laboratories, Inc.

Abode Systems Incorporated

Mitsubishi Denki Kabushiki Kaisha

Mitsubishi Electric Corporation

Butman, Moshe

Butman, Ayelet

Shmuel Avidan

Jen-Chan Jeff Chien

Chintan Intwala

Wojciech Matusik

William T Freeman

Ayelet Butman

Moshe Butman

Lubomir D Bourdev

Taeg Sang Cho

Ariel Shamir

Ariel Elbaz

Jue Wang

Matthew E Brand

Lubomira A Dontcheva

Sylvain Philippe Paris

Neel Suresh Joshi

Anant Gilra

Wilmot Wei-Mau Li

Alexandre S Parenteau

Lior Shapira

Paul E Green

Arial Shamir

Michael Rubenstein

Qiang Zhu

Kevin T Dale

Suporn Pongnumkul

Michael Rubinstein

A plurality of segment transitions in a video data stream is identified. The identifying the plurality of segment transitions includes recognizing an interface feature of a first application interface displayed in the video data stream and generating metadata associating a selected segment transition with a particular point in the video data stream. The interface feature indicates the selected segment transition among the plurality of segment transitions in the video data stream. A navigation system for navigating among the plurality of segment transitions in the video data stream is provided. Responsive to input, navigation to the selected segment transition is performed and the video data stream is displayed from the particular point.

Methods and apparatus for tutorial video enhancement

Various embodiments of a system and methods for labeling images are described. An image labeling system may receive multiple images. A subset of the images may be labeled to identify image content or elements, such as faces. The system may display some of the labeled image elements in different portions of a display area. Unlabeled image elements may be displayed in the same display area. The display size and position of each unlabeled image element may be dependent on similarities between the unlabeled image element and the displayed, labeled image elements. The system may receive input which indicates that one or more of the displayed, unlabeled image elements should receive the same label as one of the displayed, labeled image elements. Similarities between image elements may be re-analyzed, dependent on the assigned labels, and the display of unlabeled images may be updated.

System and method for labeling a collection of images

Various embodiments of a system and methods for real-time image collection and sharing are described. A group of geographically co-located mobile device users may capture images on the mobile devices during a session. The devices may send the images, during the same session, to a database where the images may be stored as a real-time image collection. The image collection may be shared, during the same session, among the mobile device users. The users may receive notification of new images in the image collection based on user preferences. The users may request, from the image collection, images containing particular content by identifying the content on an image and sending a search request which includes the identified content. A computation server may maintain a data set which associates similar images in the image collection. The data set may be updated for each new image that is added to the image collection.

System and method for real-time image collection and sharing

A method resizes input images by first constructing a grid graph. The grid graph includes one node for each pixel in the input image, and adjacent nodes in the grid graph are connected by arcs. Each arc is directed and has an associated cost. A cut is applied to the arcs of the grid graph using a cost function. A seam of pixels is determined from the cut so that coordinates of the pixels in the seam enforce monotonicity and connectivity constraints. Then, the input image is resized according to the seam to produce an output image while minimizing a change of energy in the output image when compared with the input image.

Content aware resizing of images and videos

A clustering method for high-dimensionality data includes identifying a set of nearest neighbors of a point in a multidimensional space and determining the centroid of the set of nearest neighbors, where the centroid is a member of the set of nearest neighbors. The method is then repeated using the neighbors identified around the computed centroid. In one embodiment, the method may terminate when the computed centroid becomes stationary over successive iterations. The resulting centroid may be returned as a mode of the data set. Points of the data set having common modes may be assigned to the same cluster.

Clustering techniques for large, high-dimensionality data sets

A system and method for performing content aware cropping or expansion may be applied to translate an object within an image. The object may be selected by defining an approximate bounding box of the object. The method may receive input indicating a lowest priority edge or corner of the object to be translated (e.g., using a drag operation). Respective energy values for pixels in regions of the image adjacent to a region comprising the selected object may be weighted based on their distance from the region and/or lowest priority edge/corner, on a cropping or expansion graph, and/or on the translation dimension and direction. Relative costs may be determined for seams of these regions dependent on the energy values. Low cost seams may be removed from one region and replicated in another region, and the region comprising the translated object may be combined with these resized regions in a modified image.

System and method for content aware in place translations in images

Systems and methods for performing image editing operations may divide an input image into overlapping patches and assign those patches to locations in a reconstructed output image such that visual artifacts are minimized. The methods may use belief propagation to compute a joint probability for the assignment of active patch labels to output image nodes. The computation may include an exclusivity term, steering the solution such that each patch is preferably only used once in the output image. The methods may include a pre-computation of a pruned list of candidate patches for placing next to each patch in the output image, dependent on local evidence (e.g., color, intensity, or user-driven placement) for each patch. The pre-computation may include determining groupings of patches, each forming a highly compatible loop of neighboring patches for a given candidate patch. The methods may be implemented as program instructions executable by a CPU and/or GPU.

Candidate pruning for patch transforms

A system and method for expansion and reduction of images uses an absolute value associated with each pixel of an input image (e.g., a color and/or intensity value) to determine a respective energy value for each pixel. For example, a given color or range of colors (e.g., skin tones, or other high-priority colors) may be assigned higher energy values than other colors and/or color ranges, and may be protected during image reduction and/or expansion. These energy values may be used to determine a cost associated with various seams of the image, which may represent the priority of the seams in the image. One or more low-cost seams may be identified for removal or replication to produce a resized image. The methods may be used in conjunction with an automated skin tone detector or a user interface that allows selection of one or more high priority colors or color ranges.

Seam-based reduction and expansion of images with color-weighted priority

A system and method for expansion and reduction of images uses pyramidal retargeting to reduce complexity in image resizing. An image pyramid may be generated dependent on a function of pixel data or energy values for each pixel of an input image. An image resizing solution may be developed by applying seam carving techniques to a lower-resolution version of the input image represented by the image pyramid, and may identify bands in the higher-resolution input image to be resized. A resizing operation may then be applied the bands based on local content. The resizing operation may include seam carving techniques, scaling techniques, and/or hybrid resizing techniques, and may be selectable by a user. Identified bands may be expanded to obtain a smoother solution. Different bands may be expanded by different amounts, based on local content. The number of bands and the expansion factors may be configurable based on user input.

Banded seam carving of images with pyramidal retargeting

A system and method for expansion and reduction of images in an image editing application uses the frequency of pixels values (e.g., absolute color and/or intensity values) of an input image to determine respective energy values for each pixel of the image. The energy values may vary inversely with the frequency of the pixel values. The energy values may be used to determine one or more low-cost seams of the image to be removed for an image reduction operation or replicated for an image expansion operation, where the cost of each seam is dependent on the energy values of the pixels of the seam. Determining the frequency of pixel values of the image may involve determining the number of pixels having an absolute pixel value in each of a plurality of pixel value ranges, and the absolute pixel values included in each range may be configurable by the user.

Seam carving and expansion of images with color frequency priority

A system and method for expansion and reduction of images uses a look-up table to define an arbitrary mapping of data (e.g., pixel values) representing an image to respective energy values. Each pixel value may represent an absolute color or intensity value, a difference in color or intensity values, or an average, derivative, minimum, or maximum of two or more pixel values. The energy values may then be used to determine one or more low-cost seams of the image to be removed for an image reduction operation or replicated for an image expansion operation, where the cost of each seam is dependent on the energy values of the pixels of the seam. The look-up table may be used to apply a threshold and/or cap on the energy values mapped to pixel values. The look-up table may also provide a mechanism for reconfiguring mappings, thresholds, and/or caps.

Seam-based reduction and expansion of images with table-based priority

A system and method for expansion and reduction of images uses a hybrid resizing technique combining seam carving and image scaling techniques to reduce or expand an input image. Seam carving techniques may be used to remove or add seams of the image to perform a portion of an overall resizing operation, according to a configurable seam carving tolerance. For example, the technique may only remove and/or replicate a predetermined number or percentage of the lowest-cost seams of the image, or may only remove or replicate the number of low-cost seams needed to achieve a specified percentage of the target resizing operation. The hybrid technique may apply scaling techniques to further resize the image to achieve a target size and/or ratio for the resized image. The hybrid technique may be implemented by program instructions of an image editing application, and the seam carving tolerance may be configurable by a user.

Hybrid seam carving and scaling of images with configurable carving tolerance

A system and method for expansion and reduction of images uses a hybrid resizing technique that combines seam carving and image scaling techniques to reduce or expand an image. Seam carving techniques may be used to remove or add seams having an average or total energy cost below a configurable threshold, where the cost of each seam is dependent on the energy values of the pixels of the seam. If a target size and/or ratio for the resized version of the image is not reached by removing or adding these seams, the hybrid resizing technique may apply standard or custom image scaling techniques to further reduce or expand the image to achieve the target size and/or ratio. The hybrid technique may be implemented by program instructions of an image editing application, and the cost threshold may be configurable by a user through a user interface of the image editing application.

Hybrid seam carving and scaling of images with configurable energy threshold

A system and method for expansion and reduction of images uses variable seam replication to expand an image. Seam carving techniques may be used to identify one or more low-cost seams of an input image, and these low-cost seams may be replicated to produce a resized version of the image. A different replication factor may be applied to different ones of the low-cost seams, dependent on the average or total energy value of each seam. For example, the lowest cost seam may be replicated twice as many times as the next lowest cost seam. The replication factor applied to each seam may be dependent on the number of low-cost seams identified for replication, which may be configurable by users. A configurable look-up table may map seam costs to replication factors, and may be accessed to determine a respective replication factor to be applied to each of the identified low-cost seams.

Variable seam replication in images with energy-weighted priority

Systems and methods for performing image editing operations may use patch transforms and inverse patch transforms to reconstruct output images from input images and to refine them using patch jittering such that visual artifacts are repaired. The methods may include generating one or more jittered versions of patch(es) initially assigned to nodes of the output image and using them as candidate patches for a refined image. Jittered versions of patches may be shifted by a small number of pixels in one or more directions. The number of jittered versions and amount of jittering exhibited by each may be configurable (e.g., programmatically or by a user) and/or may be dependent on the amount of overlap between the patches. Belief propagation may be used to replace patches in the output image with jittered versions in the refined image. The methods may be implemented as program instructions executable on a CPU and/or GPU.

Growing community of inventors

Brookline, MA, United States of America

Shmuel Avidan