Apple Inc.

Unity Ipr Aps

Christophe Souchard

A method of registering multiple devices to a single coordinate system is disclosed. A digital video is received from a digital camera device. The digital video presents a 2D view of a three-dimensional environment surrounding the mobile device. A set of simulated images is generated based on each video frame in the digital video. The simulated images and video frame are analyzed to determine features and create scale and orientation invariant descriptors that include depth information for the features. The descriptors and other data are sent to the server over a network. The descriptors are matched to feature descriptors from a second mobile device. A registration matrix is determined that represents a transformation from a coordinate system in the mobile device to a coordinate system in the second mobile device. The registration matrix is stored on the server to perform the registering of the multiple devices.

System and method for device synchronization in augmented reality

A system includes one or more hardware processors, a head mounted display (HMD), a digital camera device, and a path extraction engine. The path extraction engine is configured to receive two-dimensional (2D) video from the digital camera device, the 2D video presenting a 2D view of a three-dimensional (3D) object, the digital camera device moving along a camera path as the digital camera device captures the 2D video, identify a plurality of 2D features associated with the 3D object within the 2D video, track the plurality of 2D features across multiple frames of the 2D video, estimate motion in pixels associated with each 2D feature of the plurality of 2D features, using the estimated motion in the pixels, identify 2D motion caused by the digital camera device moving along the camera path, and compute a first camera position of the digital camera device based on the identified 2D motion.

Systems and methods for processing a 2D video

Some embodiments allow a video editor to spatially and temporally align two or more video sequences into a single video sequence. As used in this application, a video sequence is a set of images (e.g., a set of video frames or fields). A video sequence can be from any media, such as broadcast media or recording media (e.g., camera, film, DVD, etc.). Some embodiments are implemented in a video editing application that has a user selectable alignment operation, which when selected aligns two or more video sequences. In some embodiments, the alignment operation identifies a set of pixels in one image (i.e., a 'first' image) of a first video sequence and another image (i.e., a 'second' image) of a second video sequence. The alignment operation defines a motion function that describes the motion of the set of pixels between the first and second images. The operation then defines an objective function based on the motion function. The operation finds an optimal solution for the objective function. Based on the objective function, the operation identifies a transform, which it then applies to the first image in order to align the first image with the second image.

Spatial and temporal alignment of video sequences

Some embodiments allow a video editor to remove unwanted camera motion from a sequence of video images (e.g., video frames). Some embodiments are implemented in a video editing application. Some of these embodiments distinguish unwanted camera motion from the intended underlying motion of a camera (e.g., panning and zooming) and/or motion of objects within the video sequence.

Smoothing and/or locking operations in video editing

Some embodiments provide a method of editing images with distortion caused by a camera lens. The method identifies a set of geometries on a set of images taken by the camera lens. The method finds transformations for applying on the set of identified geometries to remove distortion on the geometries. The method applies the transformations to images taken by the camera lens to remove distortions. In some embodiments the identified geometries are curvilinear lines that correspond to straight lines in the scene being captured by the lens. Some embodiments provide a method of matching distortion for adding graphical objects to images. The method first removes the distortion on the images as described above. The method then adds the graphical objects to these images. Finally, the method applies inverse transformations on the images to restore them to their original form.

Estimating and removing distortion from an image

Estimating and removing lens distortion from scenes

An interpolation manager constructs multiple motion compensated interpolated frames between two existing frames, and then fuses the multiple interpolated frames into a single output frame. The interpolation manager constructs each of the multiple interpolated frames between the existing frames by selecting a pixel set from each existing frame, generating associated meshes, estimating flow motions in each direction, generating corresponding motion compensated meshes, computing warped images corresponding to each of the existing frames and combining the warped images into an interpolated frame. For each constructed interpolated frame, the interpolation manager uses different classification criteria to select the pixel sets, such that the pixel sets selected for each of the motion compensated interpolated frames vary, and hence the interpolated frames vary as well. The interpolation manager fuses the multiple interpolated frames into a single, output interpolated frame.

Producing smooth motion compensated frames by combining multiple interpolation results

A resampling process uses a predictive model to capture the spatial correlation among pixels in an input image, thus enabling adaptation of resampling schemes to the specific contents of the signal and noise contained in the image. The process combines automatic adjustment of a set of tap weights and formation of the inner product of a set of tap inputs and the corresponding set of tap weights to produce an output resized image.

Growing community of inventors

Marina Del Rey, CA, United States of America

Christophe Souchard