Ricoh Company, Ltd.

Other

Carnegie Mellon University

Ivana Tosic

Kathrin Berkner

Wanmin Wu

Nikhil Balram

Noah Denis Bedard

Lingfei Meng

Patrick Llull

Manuel Martinello

Liang Gao

Leonidas Spinoulas

Sofia Karygianni

Pascal Frossard

David Geoffrey Stork

M Dirk Robinson

Jelena Kovacevic

Junlan Yang

Anupama Kuruvilla

Liyang Lu

Logan Courtney

Yenting Lin

A plenoptic otoscope captures images used in making a medical diagnosis. For example, the plenoptic data can be processed to produce enhanced imagery of the ear interior, and this enhanced imagery can be used in making a medical diagnosis.

Use of plenoptic otoscope data for aiding medical diagnosis

A three-dimensional image of the eardrum is automatically registered. In one aspect, a three-dimensional image (e.g., depth map) of an eardrum is produced from four-dimensional light field data captured by a light field otoscope. The three-dimensional image is registered according to a predefined standard form. The eardrum is then classified based on the registered three-dimensional image. Registration may include compensation for out-of-plane rotation (tilt), for in-plane rotation, for center location, for translation, and/or for scaling.

Automatic eardrum registration from light field data

Eardrums are automatically classified based on a feature set from a three-dimensional image of the eardrum, such as might be derived from a plenoptic image captured by a light field otoscope. In one aspect, a grid is overlaid onto the three-dimensional image of the eardrum. The grid partitions the three-dimensional image into cells. One or more descriptors are calculated for each of the cells, and the feature set includes the calculated descriptors. Examples of descriptors include various quantities relating to the depth and/or curvature of the eardrum. In another aspect, isocontour lines (of constant depth) are calculated for the three-dimensional image of the eardrum. One or more descriptors are calculated for the isocontour lines, and the feature set includes the calculated descriptors. Examples of descriptors include various quantities characterizing the isocontour lines.

Automatic classification of eardrum shape

A procedure to calibrate a depth-disparity mapping for a plenoptic imaging system. In one aspect, one or more test objects located at known field positions and known depths are presented to the plenoptic imaging system. The plenoptic imaging system captures plenoptic images of the test objects. The plenoptic images include multiple images of the test objects captured from different viewpoints. Disparities for the test objects are calculated based on the multiple images taken from the different viewpoints. Since the field positions and depths of the test objects are known, a mapping between depth and disparity as a function of field position can be determined.

Disparity-to-depth calibration for plenoptic imaging systems

A scene is segmented into objects based on light field data for the scene, including based on image pixel values (e.g., intensity) and disparity map(s). In one aspect, the light field data is used to estimate one or more disparity maps for the scene taken from different viewpoints. The scene is then segmented into a plurality of regions that correspond to objects in the scene. Unlike other approaches, the regions can be variable-depth. In one approach, the regions are defined by boundaries. The boundaries are determined by varying the boundary to optimize an objective function for the region defined by the boundary. The objective function is based in part on a similarity function that measures a similarity of image pixel values for pixels within the boundary and also measures a similarity of disparities for pixels within the boundary.

Object segmentation from light field data

A mobile information gateway comprises: a wearable human interface module having an image delivery and display mechanism for presenting information overlaid upon a wide field of view, a computing and communication module adapted to send and receive information to and from the human interface module; and a backend service server coupled for processing medical data. The present invention also includes a method comprising: capturing information with a first mobile information gateway device; processing the captured information to determine an identity of a first user and authenticate the first user; processing the captured information to determine an identity of a patient and authenticate the patient; processing the identity of the first user and the identity of the patient to retrieve information of the patient; and resenting the retrieved information of the patient overlaid on a field of view by the first mobile information gateway device.

Mobile information gateway for use by medical personnel

A mobile information gateway comprises: a wearable human interface module having an image delivery and display mechanism for presenting information overlaid upon a wide field of view, a computing and communication module adapted receive information from the human interface module and adapted to send commands and information to the human interface module including information for presentation; and a backend service server coupled for processing data from the computing and communication module including user identification and verification. The present invention also includes a method comprising capturing information with a wearable human interface module; processing the captured information to determine an identity of a first customer; processing the identity of the first customer to retrieve information related to the first customer; and presenting the information related to the first customer with the wearable human interface module overlaid upon a field of view.

Mobile information gateway for customer identification and assignment

A system and method that identifies an object and a viewpoint from an image with a probability that satisfies a predefined criterion is described. An active view planning application receives a first image, performs recognition on the first image to determine an object, a viewpoint and a probability of recognition, determines a first expected gain in the probability of recognition when a first action is taken and a second expected gain in the probability of recognition when a second action is taken, and identifies a next action from the first action and the second action based on an increase in expected gains.

Active view planning by deep learning

A single imaging platform for making in vivo measurements of an individual's eye, where the measurements are sufficient to construct an optical model of the individual's eye. The platform includes a plenoptic opthalmic camera and an illumination module. In one configuration, the plenoptic opthalmic camera captures a plenoptic image of a corneal anterior surface of the individual's eye. In another configuration, the plenoptic opthalmic camera operates as a wavefront sensor to measure a wavefront produced by the individual's eye.

Construction of an individual eye model using a plenoptic camera

Multiframe reconstruction combines a set of acquired images into a reconstructed image. Here, which images to acquire are selected based at least in part on the content of previously acquired images. In one approach, a set of at least three images of an object are acquired at different acquisition settings. For at least one of the images in the set, the acquisition setting for the image is determined based at least in part on the content of previously acquired images. Multiframe image reconstruction, preferably via a multi-focal display, is applied to the set of acquired images to synthesize a reconstructed image of the object.

Adaptive image acquisition and display using multi-focal display

A multi-focal display represents a 3-dimensional scene by a series of 2-dimensional images located at different focal planes. The locations of the focal planes are selected based on an analysis of the three-dimensional scene to be rendered by the multi-focal display.

Content-adaptive multi-focal display

A multifocal display for rendering a 3D scene as a series of 2D images. In one aspect, the multifocal display includes a display, an optical imaging system, a refractive focus actuator and a controller. The display renders the 2D images. The optical imaging system is image-side telecentric and creates an image of the display. The refractive focus actuator is positioned at the pupil of the optical imaging system. Thus, adjusting the refractive focus actuator alters a location of the image of the display but does not significantly alter a size of the image. The controller coordinates adjustment of the refractive focus actuator with rendering of the 2D images on the display. The waveform driving the focus actuator is preferably designed to reduce ringing and jitter effects.

Multifocal display

A plenoptic camera captures a plenoptic image of an object illuminated by a point source (preferably, collimated illumination). The plenoptic image is a sampling of the four-dimensional light field reflected from the object. The plenoptic image is made up of superpixels, each of which is made up of subpixels. Each superpixel captures light from a certain region of the object (i.e., a range of x,y spatial locations) and the subpixels within a superpixel capture light propagating within a certain range of directions (i.e., a range of u,v spatial directions). Accordingly, optical properties estimation, surface normal reconstruction, depth estimation, and three-dimensional rendering can be provided by processing only a single plenoptic image. In one approach, the plenoptic image is used to estimate the bidirectional reflectance distribution function (BRDF) of the object surface.

Estimating surface properties using a plenoptic camera

A mobile information gateway comprises: a wearable human interface module having an image delivery and display mechanism for presenting information overlaid upon a wide field of view, a computing and communication module adapted to send and receive information to and from the human interface module; and a backend service server coupled for processing data. The present invention also includes a method for using a mobile information gateway comprises: capturing information with a mobile information gateway device; processing the information captured by the mobile information gateway device to detect an emergency condition; analyzing the information captured by the mobile information gateway device to identify an emergency response; retrieving instructional information corresponding the emergency response; and providing the instructional information on the mobile information gateway device.

Mobile information gateway for use in emergency situations or with special equipment

A system and method for facilitating private user interaction is disclosed. The method comprises capturing information with a first mobile information gateway device; processing the captured information to determine an identity of a first user; processing the captured information to authenticate the first user; processing the identity of the first user to retrieve information of the first user; and presenting the retrieved information of the first user overlaid upon a first field of view by the first mobile information gateway device; and presenting the retrieved information to a second user overlaid upon a second field of view by the second mobile information gateway device.

Growing community of inventors

San Francisco, CA, United States of America

Ivana Tosic