Intuitive Surgical Operations, Inc.

Magic Leap, Inc.

Sarnoff Corporation

Scanadu, Inc.

Daqri, LLC

Healthy.io Ltd

Imperial Innovations Limited

Wenyi Zhao

Tao Zhao

Brian David Hoffman

Brandon D Itkowitz

Catherine J Mohr

Christopher J Hasser

Simon Peter DiMaio

David D Scott

David Q Larkin

Paul W Mohr

Ian E McDowall

Jeffrey M DiCarlo

Daniel J Halabe

William C Nowlin

Simon P Dimaio

Paul E Lilagan

John D Stern

Theodore W Rogers

David W Robinson

Craig R Gerbi

Chenyu Wu

Walter Adolf De Brouwer

David Hirvonen

Samuel A Miller

Lionel Ernest Edwin

Amy E Kerdok

Etienne Gregoire Grossmann

Michael Robert Johnson

Jonathan Michael Sorger

Stephen Charles Hsu

Brian Edward Miller

Martin Zizi

Christopher Broaddus

David Nister

Keith James Hanna

Brian Mullins

Rakesh Kumar

Supun Samarasekera

Denny SzuPao Liao

Guang-Zhong Yang

Simon Di Maio

Brandon D Woolsey

Whitney Morgan McGowan

Nukul Sanjay Shah

Po-Kang Huang

Brian Christopher Clark

Eron Anthony Villarreal

Hubert Stein

Richmond B Chan

Bing Wu

Myrian J Curet

David James Hirvonen

Myriam J Curet

Richard Patrick Wildes

Pip Tompkin

Aniket Murarka

Yves A Behar

George Mylonas

Chao Zhang

Lawrence Arnold Gunn

Samuel McClellan

Ali M Tatari

Saud Akram

Madison Smith

Muzaffer Kal

Varun Nasery

Brendan Drew

Chen-Chi Chu

Gung-Zhong Yang

Hua Hui

Ian McDowell

Christopher J J Hasser

Bodgan Matei

William Brian Lehman

Thomas Edward Klinedinst

Barbara Levienaise-Obadia

Ian M Mcdowall

An exemplary method includes receiving a frame of raw pixels captured by an image capture sensor, a particular location in the frame of raw pixels including a captured color pixel value of a first color component of a first color space; generating, for the particular location in the frame of raw pixels, a vertical color difference signal and a horizontal color difference signal using information from the frame of raw pixels; determining that the particular location is missing a color pixel value of a second color component of the first color space; generating a reconstructed color pixel value of the second color component based on the vertical color difference signal and the horizontal color difference signal; and transforming the vertical color difference signal and the horizontal color difference signal into chrominance pixel values in a second color space that is different from the first color space.

Efficient image demosaicing and local contrast enhancement

A system comprises a first robotic arm adapted to support and move a tool and a second robotic arm adapted to support and move a camera configured to capture an image of a camera field of view. The system further comprises an input device, a display, and a processor. The processor is configured to display a first synthetic image including a first synthetic image of the tool. The first synthetic image of the tool includes a portion of the tool outside of the camera field of view. The processor is also configured to receive a user input at the input device and responsive to the user input, change the display of the first synthetic image to a display of a second synthetic image including a second synthetic image of the tool that is different from the first synthetic image of the tool.

Synthetic representation of a surgical robot

A display system can include a head-mounted display configured to project light to an eye of a user to display virtual image content at different amounts of divergence and collimation. The display system can include an inward-facing imaging system possibly comprising a plurality of cameras that image the user's eye and glints thereon and processing electronics that are in communication with the inward-facing imaging system and that are configured to obtain an estimate of a center of cornea of the user's eye using data derived from the glint images. The display system may use spherical and aspheric cornea models to estimate a location of the corneal center of the user's eye.

Eye tracking using aspheric cornea model

A system comprises: a robotic arm operatively coupleable to a tool comprising a working end; and an input device communicatively coupled to the robotic arm. The input device is manipulatable by an operator. The system further comprises a processor configured to cause an image of a work site, captured by an image capture device from a perspective of an image reference frame, to be displayed on a display. The image of the work site includes an image of the working end of the tool. The processor is further configured to determine a position of the working end of the tool in the image of the work site and render a tool information overlay at the position of the working end of the tool in the image of the work site. The tool information overlay visually indicates an identity of the input device.

Rendering tool information as graphic overlays on displayed images of tools

A system may comprise a tool including at least one reference feature. a processor, and a memory having computer readable instructions stored thereon. The computer readable instructions, when executed by the processor, may cause the system to receive image data including an image of the tool and the at least one reference feature, determine a pose of the tool from the image data, and modify the image data to visually decrement a portion of the image data corresponding to the at least one reference feature.

Configuration marker design and detection for instrument tracking

An example method includes causing an illuminator to simultaneously illuminate tissue with a first visible color illumination component, a second visible color illumination component, and a fluorescence excitation illumination component; and causing a display unit to display, based on the illumination, a display scene that includes: a reduced color scene component corresponding to the first and second visible color illumination components, and a highlighted scene component corresponding to the fluorescence excitation illumination component.

Multi-function imaging

The systems and methods described can include approaches to calibrate head-mounted displays for improved viewing experiences. Some methods include receiving data of a first target image associated with an undeformed state of a first eyepiece of a head-mounted display device; receiving data of a first captured image associated with deformed state of the first eyepiece of the head-mounted display device; determining a first transformation that maps the first captured image to the image; and applying the first transformation to a subsequent image for viewing on the first eyepiece of the head-mounted display device.

Compensation for deformation in head mounted display systems

A display system can include a head-mounted display configured to project light to an eye of a user to display virtual image content at different amounts of divergence and collimation. The display system can include an inward-facing imaging system possibly comprising a plurality of cameras that image the user's eye and glints thereon, and processing electronics that are in communication with the inward-facing imaging system and that are configured to obtain an estimate of a center of cornea of the user's eye using data derived from the glint images. The display system may use spherical and aspheric cornea models to estimate a location of the corneal center of the user's eye.

A medical robotic system includes a viewer for displaying an image of a work site, a gaze tracker for tracking a gaze point of a user on the viewer, and a processor. The processor is configured to: draw an area or volume defining shape, overlaid on the image of the work site, in a position determined by the gaze tracker; assign a fixed virtual constraint to the shape and constrain movement of a robotic tool according to the fixed virtual constraint; receive a user selected action command selecting an image of patient anatomy; and superimpose the selected image of the patient anatomy over the image of the work site within the shape. The selected image of the patient anatomy is registered to the image of the work site.

Robotic system providing user selectable actions associated with gaze tracking

A robotic system may comprise a first robotic arm operatively coupleable to a first tool. The first tool has a first working end. The system may also comprise an image capture device, a display, and a processor. The processor may be configured to cause an image of a work site, which was captured by the image capture device from a perspective of an image reference frame, to be displayed on the display. The image of the work site includes an image of the first working end of the first tool. The processor may also determine a position of the first working end of the first tool in the image of the work site and render a tool information overlay at the position of the first working end of the first tool in the image of the work site. The tool information overlay visually indicates state information for the first tool. The processor may also change the tool information overlay while the first tool is in a first operational state by changing a brightness of the tool information overlay.

A calibration assembly may be used with an endoscopic imaging system having a field of view. The calibration assembly may comprise an interface configured for constraining engagement with the endoscopic imaging system and a target coupled with the interface so as to be within the field of view. The target may include a plurality of calibration features. The calibration assembly may be reconfigurable from a first spatial arrangement between the target and the endoscopic imaging system to a second spatial arrangement between the target and the endoscopic imaging system. The calibration assembly may also comprise a processor configured to obtain, from the endoscopic imaging system, first and second images of at least some of the calibration features of the target at the first and second spatial arrangements. The processor may also generate calibration data for the endoscopic imaging system using the first and second images.

Targets, fixtures, and workflows for calibrating an endoscopic camera

An exemplary method includes a demosaic module receiving a frame of raw pixels having values in a first color space, generating vertical and horizontal color difference signals using information from the frame of raw pixels, generating reconstructed color pixel values for raw pixels that are missing captured color pixel values of a first color component of the first color space, transforming the captured and reconstructed color pixel values of the first color component to brightness pixel values in a second color space, and transforming the vertical and horizontal color difference signals into chrominance pixel values in the second color space.

A system may comprise an image capture device to capture an image of a work site. The system may also comprise a processor to determine whether a tool disposed at the work site is energized and determine a first area of the captured image of the work site, including or adjacent to an image of a portion of the tool in the captured image. The processor may also determine a second area of the captured image including a remainder of the captured image that does not include the first area. Conditioned upon determining that the tool is energized, at least one of the first area and the second area of the captured image of the work site may be processed to indicate that the tool in the first area is being energized. The image of the work site with the first area and the second area may be displayed.

Providing information of tools by filtering image areas adjacent to or on displayed images of the tools

An example method includes receiving a single frame comprising a first set of pixel data that includes a fluorescence scene component and a second set of pixel data that includes a combination of a visible color component scene and the fluorescence scene component; and generating, based on the first set of pixel data that includes the fluorescence scene component and the second set of pixel data that includes the combination of the visible color component scene and the fluorescence scene component, a display scene.

Growing community of inventors

Mountain View, CA, United States of America

Wenyi Zhao