The Johns Hopkins University

International Business Machines Corporation

Intuitive Surgical Operations, Inc.

Integrated Surgical Systems, Inc.

Peter Kazanzides

Russell H Taylor

Ehsan Azimi

Long Qian

Anton Deguet

David Q Larkin

Simon Peter DiMaio

Christopher J Hasser

Nassir Navab

Brent D Mittelstadt

Joshua Leven

Marcin Arkadiusz Balicki

Bela L Musits

William A Hanson

Balazs Peter Vagvolgyi

Howard A Paul

Edward Glassman

Andre Pierre Gueziec

Emad M Boctor

Ralph Etienne-Cummings

Willie Williamson, Jr

Muyinatu Bell

Alind Sahay

Xia Tian

Bálazs Peter Vágvölgyi

Joel Zuhars

Ndubuisi John Ekekwe

Camilo Molina

Judy Huang

A computer-implemented method includes: receiving, by an augmented reality device, a medical image of a surgical site, generating, by the augmented reality device, a virtual surgical site model based on the medical image; presenting, by the augmented reality device, the virtual surgical site model; receiving, by the augmented reality device, user calibration input; aligning, by the augmented reality device, the virtual surgical site model with a real-life surgical site based on the user calibration input; and displaying, by the augmented reality device and after the aligning, a virtual insertion path between an incision point and a target point to aid in inserting a tool as part of performing a surgical procedure.

Augmented reality based surgical navigation system

A calibration platform may obtain measurements for aligning a real-world coordinate system and a display coordinate system. For example, the calibration platform may display, via an optical see-through head-mounted display (OST-HMD), a three-dimensional virtual object and receive, from a positional tracking device, information that relates to a current pose of a three-dimensional real-world object to be aligned with the three-dimensional virtual object. The calibration platform may record a three-dimensional position of a plurality of points on the three-dimensional real-world object based on the current pose of the three-dimensional real-world object, based on an indication that the plurality of points on the three-dimensional real-world object respectively corresponds with a plurality of points on the three-dimensional virtual object. Accordingly, based on the obtained measurements, the calibration platform may generate a transformation function to provide a mapping between three-dimensional points in the real-world coordinate system and three-dimensional points in the display coordinate system.

Calibration system and method to align a 3D virtual scene and a 3D real world for a stereoscopic head-mounted display

A calibration platform may display, via an optical see-through head-mounted display (OST-HMD), a virtual image having at least one feature. The calibration platform may determine, based on information relating to a gaze of a user wearing the OST-HMD, that the user performed a voluntary eye blink to indicate that the at least one feature of the virtual image appears to the user to be aligned with at least one point on the three-dimensional real-world object. The calibration platform may record an alignment measurement based on a position of the at least one point on the three-dimensional real-world object in a real-world coordinate system based on a time when the user performed the voluntary eye blink. Accordingly, the alignment measurement may be used to generate a function providing a mapping between three-dimensional points in the real-world coordinate system and corresponding points in a display space of the OST-HMD.

Blink-based calibration of an optical see-through head-mounted display

In one embodiment of the invention, a a minimally invasive surgical system is disclosed. The system configured to capture and display camera images of a surgical site on at least one display device at a surgeon console; switch out of a following mode and into a masters-as-mice (MaM) mode; overlay a graphical user interface (GUI) including an interactive graphical object onto the camera images; and render a pointer within the camera images for user interactive control. In the following mode, the input devices of the surgeon console may couple motion into surgical instruments. In the MaM mode, the input devices interact with the GUI and interactive graphical objects. The pointer is manipulated in three dimensions by input devices having at least three degrees of freedom. Interactive graphical objects are related to physical objects in the surgical site or a function thereof and are manipulatable by the input devices.

Interactive user interfaces for minimally invasive telesurgical systems

A device may determine a view of a user of a head mounted display. The device may obtain tracking data relating to a surgical procedure. The device may obtain, from an imaging device, surgical imaging relating to the surgical procedure. The device may orient models of objects based on the tracking data and the view of the user of the head mounted display, wherein the objects includes the imaging device. The device may augment, by providing output to the head mounted display for display, the view of the user with contextual information relating to the objects based on orienting the models based on the tracking data and the view of the user, wherein the contextual information includes the surgical imaging captured by an imaging device.

Augmented reality display for surgical procedures

The present invention is directed to a method and system for photoacoustic imaging for guiding medical procedures. A transducer is placed near the site of the procedure. The optical fiber, coupled to an electromagnetic source, such as a laser, is attached to a medical device. During the procedure, the device and optical fiber are inserted into the procedure site where the optical fiber illuminates the procedure site, which has a thickness of approximately 2 mm. Photoacoustic images are acquired to visualize the procedure site as the procedure is proceeding in order to provide real-time guidance. This system is applicable to multiple surgical and interventional procedures, such as transsphenoidal surgery.

Method and system for transcranial photoacoustic imaging for guiding skull base surgeries

According to some embodiments of the present invention, a cooperatively controlled robot includes a robotic actuator assembly comprising a tool holder and a force sensor, a control system adapted to communicate with the robotic actuator assembly and the force sensor, and an output system in communication with the control system. The tool holder is configured to receive a tool to be manipulated by a user. The control system is configured to receive an instruction from a user to switch from a robot control mode into a user interface control mode. The force sensor is configured to measure at least one of a force and a torque applied to the tool, and the control system is configured to receive an indication of the at least one of a force and a torque applied to the tool and manipulate the output system based on the indication.

Surgical system user interface using cooperatively-controlled robot

In one embodiment of the invention, a method for a minimally invasive surgical system is disclosed. The method includes capturing and displaying camera images of a surgical site on at least one display device at a surgeon console; switching out of a following mode and into a masters-as-mice (MaM) mode; overlaying a graphical user interface (GUI) including an interactive graphical object onto the camera images; and rendering a pointer within the camera images for user interactive control. In the following mode, the input devices of the surgeon console may couple motion into surgical instruments. In the MaM mode, the input devices interact with the GUI and interactive graphical objects. The pointer is manipulated in three dimensions by input devices having at least three degrees of freedom. Interactive graphical objects are related to physical objects in the surgical site or a function thereof and are manipulatable by the input devices.

Systems and methods for interactive user interfaces for robotic minimally invasive surgical systems

A combined teleoperative-cooperative controllable robotic system includes a robotic actuator assembly, a control system adapted to communicate with the robotic actuator assembly, and a teleoperation unit adapted to communicate with the control system. The control system is configured to control at least a first portion of the robotic actuator assembly in response to at least one of a force or a torque applied to at least a second portion of the robotic actuator assembly by a first user for cooperative control. The control system is further configured to control at least a third portion of the robotic actuator assembly in response to input by a second user from the teleoperation unit for teleoperative control.

Teleoperative-cooperative robotic system

Interactive user interfaces for robotic minimally invasive surgical systems

An integrated circuit for controlling a DC motor is disclosed. The integrated circuit includes at least one digital position and speed circuit (DPS) for providing measurements of speed, position, and direction of the motor, the DPS being in signal communication with the motor for receiving a pair of signals having a quadrature relationship; and at least one programmable gain amplifier (PGA) electrically coupled to the motor, the PGA being configured to receive a feedback signal indicative of current flowing through the motor and to apply a second signal to the motor for adjusting the speed of the motor; and at least two analog-to-digital converters (A/D), one A/D being used to quantize the output of the PGA for an off-chip processor; and another A/D to provide motor reference position from an analog sensor, such as a potentiometer; and at least two digital-to-analog converters (D/A), one D/A used to set the motor voltage; and another D/A used to set the motor current limit. The integrated circuit can be incorporated into a larger motor control loop which further includes a summing amplifier for providing the feedback signal to the motor that is indicative of current flowing through the motor; a buffer amplifier electrically for sensing the output current of the motor, and a processor for providing control signals to the system monolithic module and for receiving the measurements of speed, position, and direction of the motor.

Adaptive and reconfigurable system for DC motor control

A method and system is disclosed for registering two dimensional fluoroscopic images with a three dimensional model of a surgical tissue of interest. The method includes steps of: (a) generating, from CT or MRI data, a three dimensional model of a surgical tissue of interest; (b) obtaining at least two, two dimensional, preferably fluoroscopic, x-ray images representing at least two views of the surgical tissue of interest, the images containing radio-opaque markers for associating an image coordinate system with a surgical (robot) coordinate system; (c) detecting the presence of contours of the surgical tissue of interest in each of the at least two views; (d) deriving bundles of three dimensional lines that pass through the detected contours; and (e) interactively matching three dimensional points on three dimensional silhouette curves obtained from the three dimensional model with the bundles of three dimensional lines until the three dimensional model is registered within the surgical coordinate system to a predetermined level of accuracy. The step of iteratively matching includes steps of: defining a distance between surfaces of the model and a beam of three dimensional lines that approach the surfaces; and finding a pose of the surfaces that minimizes a distance to the lines using, preferably, a statistically robust method, thereby providing a desired registration between a surgical robot and a preoperative treatment plan.

Methods and apparatus for registering CT-scan data to multiple

Methods, systems and apparatus for planning the position of a prosthesis in a long bone in orthopaedic surgical procedures, such as hip replacement surgery, knee replacement surgery, long bone osteotomies, and the like. A bone model is generated from a scanned image of a bone, a prosthesis model is selected from a library of prosthesis models and then a cavity model is formed based on the prosthesis model and/or the bone model. The cavity model may then be positioned over the bone model, either interactively by the surgeon or automatically through an algorithm based on clinical parameters, to determine a reasonable location for implantation of a prosthesis within the bone. The cavity model allows the surgeon to optimize placement of the implant within the bone, and it provides important clinical information to the surgeon, such as areas in which press fits are provided, extension areas for possible subsidence and access areas for allowing the surgeon to insert the implant into the cavity.

System and method for cavity generation for surgical planning and

A robotic surgical system (10) includes a multiple degree of freedom manipulator arm (14) having a surgical tool (22). The arm is coupled to a controller (24) for controllably positioning the surgical tool within a three dimensional coordinate system. The system further includes a safety monitoring processor (38) for determining the position of the surgical tool in the three dimensional coordinate system relative to a volumetric model. The volumetric model may be represented as a constructive solid geometry (CSG) tree data structure. The system further includes an optical tracking camera system (28,32) disposed for imaging a region of space that includes at least a portion of the manipulator arm. An output of the camera system is coupled to the processor (38) that processes the volumetric model for determining if the surgical tool is positioned outside of the volumetric model. The system further includes a strain gage (40) for detecting slippage in three dimensions between an immobilized tissue, such as bone, and a reference point (44). The system also includes multiple and redundant safety features for suspending a motion of the surgical tool to prevent the tool from operating outside of the predetermined volume of space.

Growing community of inventors

Towson, MD, United States of America

Peter Kazanzides