Google Inc.

Disney Enterprises, Inc.

Irfan Essa

Thad Eugene Starner

Matthias Grundmann

Vivek Kwatra

Krishna Asur Bharat

Caroline Rebecca Pantofaru

Vinay Bettadapura

Hayes Solos Raffle

Weilong Yang

Daniel M Aminzade

Honglak Lee

Tianhao Zhang

Hung-Yu Tseng

Lu Jiang

Anelia Angelova

Iain Matthews

Jessica Kate Hodgins

Rahul Sukthankar

Ariel Shamir

Steven Hickson

Kihwan Kim

A method for generating an output image from an input image and an input text instruction that specifies a location and a modification of an edit applied to the input image using a neural network is described. The neural network includes an image encoder, an image decoder, and an instruction attention network. The method includes receiving the input image and the input text instruction; extracting, from the input image, an input image feature that represents features of the input image using the image encoder; generating a spatial feature and a modification feature from the input text instruction using the instruction attention network; generating an edited image feature from the input image feature, the spatial feature and the modification feature; and generating the output image from the edited image feature using the image decoder.

Image manipulation by text instruction

A method for personalizing a content item using captured footage is disclosed. The method includes receiving a first video feed from a first camera, wherein the first camera is designated as a source camera for capturing an event during a first time duration. The method also includes receiving data from a second camera, and determining, based on the received data from the second camera, that an action was performed using the second camera, the action being indicative of a region of interest (ROI) of the user of the second camera occurring within a second time duration. The method further includes designating the second camera as the source camera for capturing the event during the second time duration.

Systems and methods for directing content generation using a first-person point-of-view device

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining a clustering of images into a plurality of semantic categories. In one aspect, a method comprises: training a categorization neural network, comprising, at each of a plurality of iterations: processing an image depicting an object using the categorization neural network to generate (i) a current prediction for whether the image depicts an object or a background region, and (ii) a current embedding of the image; determining a plurality of current cluster centers based on the current values of the categorization neural network parameters, wherein each cluster center represents a respective semantic category; and determining a gradient of an objective function that includes a classification loss and a clustering loss, wherein the clustering loss depends on a similarity between the current embedding of the image and the current cluster centers.

Category learning neural networks

Methods and devices for initiating a search of an object are disclosed. In one embodiment, a method is disclosed that includes receiving video data recorded by a camera on a wearable computing device, where the video data comprises at least a first frame and a second frame. The method further includes, based on the video data, detecting an area in the first frame that is at least partially bounded by a pointing device and, based on the video data, detecting in the second frame that the area is at least partially occluded by the pointing device. The method still further includes initiating a search on the area.

Object occlusion to initiate a visual search

A method for localizing the attention of a user of a first-person point-of-view (FPPOV) device is disclosed. The method includes receiving data from an FPPOV device, the data being indicative of a first region-of-interest (ROI) of an event for a first time duration and a second ROI of the event for a second time duration. The method further include determining that a first camera from a plurality of cameras best captures the first ROI during the first time duration, and determining that a second camera from the plurality of cameras best captures the second ROI during the second time duration.

An easy-to-use online video stabilization system and methods for its use are described. Videos are stabilized after capture, and therefore the stabilization works on all forms of video footage including both legacy video and freshly captured video. In one implementation, the video stabilization system is fully automatic, requiring no input or parameter settings by the user other than the video itself. The video stabilization system uses a cascaded motion model to choose the correction that is applied to different frames of a video. In various implementations, the video stabilization system is capable of detecting and correcting high frequency jitter artifacts, low frequency shake artifacts, rolling shutter artifacts, significant foreground motion, poor lighting, scene cuts, and both long and short videos.

Cascaded camera motion estimation, rolling shutter detection, and camera shake detection for video stabilization

Described herein are methods, systems, apparatuses and products for utilizing motion fields to predict evolution in dynamic scenes. One aspect provides for accessing active object position data including positioning information of a plurality of individual active objects; extracting a plurality of individual active object motions from the active object position data; constructing a motion field using the plurality of individual active object motions; and using the motion field to predict one or more points of convergence at one or more spatial locations that active objects are proceeding towards at a future point in time. Other embodiments are disclosed.

System and method for utilizing motion fields to predict evolution in dynamic scenes

A method for localizing the attention of a user of a first-person point-of-view (FPPOV) device is disclosed. The method includes receiving a plurality of images of an event, each image having been captured by one of a plurality of reference cameras during a first time duration. The method further includes receiving a first user-captured image captured by the FPPOV device during the first time duration. A first image of the plurality of images is selected as a best-matched image, based on the first user-captured image, for capturing a region-of-interest.

Systems and methods for attention localization using a first-person point-of-view device

Methods and systems for processing a video for stabilization are described. A recorded video may be stabilized by removing at least a portion of shake introduced in the video. An original camera path for a camera used to record the video may be determined. A crop window size may be selected, a crop window transform may accordingly be determined, and the crop window transform may be applied to the original video to provide a modified video from a viewpoint of the modified motion camera path.

Methods and systems for processing a video for stabilization using dynamic crop

Methods and devices for initiating a search of an object are disclosed. In one embodiment, a method is disclosed that includes receiving sensor data from a sensor on a wearable computing device and, based on the sensor data, detecting a movement that defines an outline of an area in the sensor data. The method further includes identifying an object that is located in the area and initiating a search on the object. In another embodiment, a server is disclosed that includes an interface configured to receive sensor data from a sensor on a wearable computing device, at least one processor, and data storage comprising instructions executable by the at least one processor to detect, based on the sensor data, a movement that defines an outline of an area in the sensor data, identify an object that is located in the area, and initiate a search on the object.

Object outlining to initiate a visual search

Methods and devices for initiating a search are disclosed. In one embodiment, a method is disclosed that includes causing a camera on a wearable computing device to record video data, segmenting the video data into a number of layers and, based on the video data, detecting that a pointing object is in proximity to a first layer. The method further includes initiating a first search on the first layer. In another embodiment, a wearable computing device is disclosed that includes a camera configured to record video data, a processor, and data storage comprising instructions executable by the processor to segment the video data into a number of layers and, based on the video data, detect that a pointing object is in proximity to a first layer. The instructions are further executable by the processor to initiate a first search on the first layer.

Using visual layers to aid in initiating a visual search

Methods and systems for rolling shutter removal are described. A computing device may be configured to determine, in a frame of a video, distinguishable features. The frame may include sets of pixels captured asynchronously. The computing device may be configured to determine for a pixel representing a feature in the frame, a corresponding pixel representing the feature in a consecutive frame; and determine, for a set of pixels including the pixel in the frame, a projective transform that may represent motion of the camera. The computing device may be configured to determine, for the set of pixels in the frame, a mixture transform based on a combination of the projective transform and respective projective transforms determined for other sets of pixels. Accordingly, the computing device may be configured to estimate a motion path of the camera to account for distortion associated with the asynchronous capturing of the sets of pixels.

Growing community of inventors

Atlanta, GA, United States of America

Irfan Essa