Google Inc.

Visbit Inc.

Microsoft Technology Licensing, LLC

Columbia University

Changyin Zhou

Marc Stewart Levoy

Samuel William Hasinoff

Baining Guo

Stephen S Lin

Roman Lewkow

Jiawen Chen

Shree Kumar Nayar

Ryan M Geiss

Peyman Milanfar

Oliver S Cossairt

Daniel Miau

Imaging systems can often gather higher quality information about a field of view than the unaided human eye. For example, telescopes may magnify very distant objects, microscopes may magnify very small objects, and high frame-rate cameras may capture fast motion. The present disclosure includes devices and methods that provide real-time vision enhancement without the delay of replaying from storage media. The disclosed devices and methods may include a live view user interface with two or more interactive features or effects that may be controllable in real-time. Specifically, the disclosed devices and methods may include a live view display and image and other information enhancements, which utilize in-line computation and constant control.

Graphic interface for real-time vision enhancement

The present disclosure relates to systems and methods for streaming a tiled representation of video content based on a viewing direction and desired zoom level. Example methods include receiving information indicative of a view direction and a zoom level. The view direction and the zoom level correspond to a field of view including a first portion of the video content. A second portion of the video content is outside of the field of view. The method also includes selecting, from a plurality of viewports, an active viewport corresponding to the first portion of the video content. The active viewport specifies a set of one or more active tiles of a tiled representation of the video content. The method additionally includes comparing the set of one or more active tiles with a set of one or more previous active tiles and causing streaming and rendering to stop with respect to previous active tiles that are not in the set of one or more active tiles. The method includes causing streaming and rendering to commence with respect to active tiles that are not in the set of one or more previously active tiles.

System, algorithms, and designs of view-optimized zoom for 360 degree video

The present disclosure relates to imaging systems and methods that include a plurality of cameras configured to capture video image data based on respective fields of view of an environment. Each camera of the plurality of cameras is communicatively coupled to neighbor cameras of the plurality of cameras via a communication interface. Each camera may carry out operations include capturing video image data of the respective field of view and determining an overlay region. The overlay region includes an overlapping portion of video image data captured by the respective camera and at least one of the neighbor cameras. The operations also include cropping and warping the captured video image data of the respective field of view based on the overlay region to form respective processed video image data. The processed video image data may be uploaded to a cloud server and provided, via multiple trunk links, to a client device.

Virtual reality 360-degree video camera system for live streaming

Systems, methods, and media for providing interactive refocusing are provided, the systems comprising: a lens; an image sensor; and a processor that: causes the image sensor to capture a plurality of images over a predetermined period of time, wherein each of the plurality of images represents a scene at a different point in time; changes a depth of field between at least a pair of the plurality of images; concatenates the plurality of images to create a duration focal volume in the order in which the images were captured; computes a space-time in-focus image that represents in-focus portions from each of the plurality of images based on the duration focal volume; and computes a space-time index map that identifies an in-focus image for each location of the scene from among the plurality of images based on the duration focal volume and the space-time in-focus image.

Systems, methods, and media for providing interactive refocusing in images

The present disclosure relates to systems and methods that employ a mechanical method for adjusting camera field angles at a high speed. The adjustments may be synchronized to a high speed image capture process. As such, multiple frames can be captured without experiencing significant object movement or hand shake. The systems and methods may be used to capture videos and/or photo stitching, because both camera position and objects in the scene are static during the high speed image capture.

Time multiplexing programmable field of view imaging

The present disclosure relates to systems and methods that deliver 360&#xb0; video streams, which may be used in virtual reality and augmented reality content. Namely, a 360&#xb0; video stream may be divided into a plurality of video streams, each stream of which corresponds to a portion of a viewing volume or viewing sphere. Additionally, the original 360&#xb0; video stream may be downsampled to a lower resolution 360&#xb0; video stream. For a given user view angle, one or more of the plurality of video streams may be considered active streams and connected to a user interface or display for data download. In such a scenario, a user interface may be connected to at least one high resolution video stream that corresponds to the user view angle as well as a low resolution 360&#xb0; video stream.

View-aware 360 degree video streaming

A plurality of images of a scene may be obtained. These images may have been captured by an image sensor, and may include a first image and a second image. A particular gain may have been applied to the first image. An effective color temperature and a brightness of a first pixel in the first image may be determined, and a mapping between pixel characteristics and noise deviation of the image sensor may be selected. The pixel characteristics may include pixel brightness. The selected mapping may be used to map at least the brightness of the first pixel to a particular noise deviation. The brightness of the first pixel and the particular noise deviation may be compared to a brightness of a second pixel of the second image. The comparison may be used to determine whether to merge the first pixel and the second pixel.

Noise models for image processing

Removal of the effects of dust or other impurities on image data is described. In one example, a model of artifact formation from sensor dust is determined. From the model of artifact formation, contextual information in the image and a color consistency constraint may be applied on the dust to remove the dust artifacts. Artifacts may also be removed from multiple images from the same or different cameras or camera settings.

Removal of image artifacts from sensor dust

Systems, methods, and media for recording an image of a scene ate provided. In accordance with some embodiments, systems for recording an image of a scene are provided. comprising. a diffuser {hat diffuses light representing the scene and that has a scattering function that is independent of aperture coordinates, a sensor that receives diffused light representing the scene and generates data representing an image; and a hardware processes that uses a point spread function to deblur the image.

Systems, methods, and media for recording an image using an optical diffuser

A method is provided that includes operating a first camera to capture a first image stream and operating a second camera to capture a second image stream. The method further includes initially using the first image stream to display a first field of view in a live-view interface of a graphic display and, while displaying the first image stream in the live-view interface, receiving an input corresponding to a zoom command. The method further includes, in response to receiving the input: (a) switching from using the first image stream to display the first field of view in the live-view interface to using a combination of the first image stream and the second stream to display a transitional field of view of the environment in the live-view interface and (b) subsequently switching to using the second image stream to display the second field of view in the live-view interface.

Systems and methods for image zoom

An example composite imaging system includes a first imaging system further comprising a first image sensor and a first aperture anterior to the first image sensor. The first aperture (i) is defined by an inner perimeter and an outer perimeter and (ii) defines, at least in part, a field of view of the first imaging system. The first imaging system further includes a plurality of reflectors posterior to the first aperture that are configured to redirect light that crosses the first aperture to the first image sensor. The first image sensor, the first aperture, and the plurality of reflectors are arranged around a common optical axis. The composite imaging system further includes a second imaging system arranged substantially anterior to the first imaging system. The second imaging system is arranged such that the second imaging system is outside a field of view of the first imaging system.

Multi-aperture imaging systems

The present disclosure provides example methods operable by computing device. An example method can include receiving an image from a camera. The method can also include comparing one or more parameters of the image with one or more control parameters, where the one or more control parameters comprise information indicative of an image from a substantially unobstructed camera. Based on the comparison, the method can also include determining a score between the one or more parameters of the image and the one or more control parameters. The method can also include accumulating, by a computing device, a count of a number of times the determined score image exceeds a first threshold. Based on the count exceeding a second threshold, the method can also include determining that the camera is at least partially obstructed.

Camera obstruction detection

Imaging systems can often gather higher quality information about a field of view than the unaided human eye. For example, telescopes may magnify very distant objects, microscopes may magnify very small objects, and high frame-rate cameras may capture fast motion. The present disclosure includes devices and methods that provide real-time vision enhancement without the delay of replaying from storage media. The disclosed devices and methods may include a live view display and image and other information enhancements, which utilize in-line computation and constant control. The disclosure includes techniques for enabling push-button slow motion effects through buffer management and the adjustment of a display frame rate.

Growing community of inventors

Mountain View, CA, United States of America

Changyin Zhou