Verily Life Sciences LLC

Martin Habbecke

Joëlle K Barral

Xing Jin

Lin Yang

Joëlle Barral

Caitlin Q Donhowe

Michal Levin

Thomas Teisseyre

Gianni Campion

Robin Thellend

Antons Krumins

Daniele Paolo Piponi

James J Shuma

Eric B Johnson

Francois Brahic

James Shuma

Miles Eldon

Neil Inala

Timothy Chung

Michael Newton

Brian Carnes

A system includes a display, and a database including surgical videos, images of organs in a human body obtained from a medical imaging device, and images of disease in a human body obtained from the medical imaging device. A controller including a processor is coupled to memory, the database, and the display, and the memory stores information that when executed by the processor causes the system to perform operations. For example, the processor may determine first organ information from the images of the organs, and first disease information from the images of the disease. The processor my calculate a similarity score between the first organ information and the first disease information and second disease information and second organ information indexed to the surgical videos. The processor selects one or more of the surgical videos based on the similarity score, and displays the surgical videos on the display.

Surgical video retrieval based on preoperative images

A surgical simulator for simulating a surgical scenario comprises a display system, a user interface, and a controller. The controller includes one or more processors coupled to memory that stores instructions that when executed cause the system to perform operations. The operations include generating simulated surgical videos, each representative of the surgical scenario. The operations further include associating simulated ground truth data from the simulation with the simulated surgical videos. The ground truth data corresponds to context information of at least one of a simulated surgical instrument, a simulated anatomical region, a simulated surgical task, or a simulated action. The operations further include annotating features of the simulated surgical videos based, at least in part, on the simulated ground truth data for training a machine learning model.

Surgical simulator providing labeled data

Systems and methods for depth detection and virtual modeling using surgical robots during a surgical procedure are described. The robotic surgical systems include robotic arms with interchangeable surgical tools. An endoscope at the end of one of the robotic arms includes a depth sensor for detecting a distance from the camera to patient anatomy. The depth data and image data are used to generate a feature model of the patient anatomy for reference by a surgeon. The feature model is displayed to the surgeon in an augmented reality view with preoperative and intraoperative images mapped thereon for reference and guidance of a surgical procedure.

Robotic surgery depth detection and modeling

Systems and methods for identifying surgery tools of a robotic surgery system in surgery video based on robot kinematics are described. Kinematic data describing a position of a first robotic surgery arm including a surgery tool is used to determine a nominal position of the first robotic surgery arm. A search area within image data obtained during a robotic surgery from an endoscope attached to a second robotic surgery arm is determined based on the kinematic data and a nominal position of the surgery tool. The surgery tool is identified within the search area of the image data using an object detection technique.

Surgery tool segmentation with robot kinematics

Examples of systems and methods for detecting delays during a surgical procedure are disclosed. For example, one disclosed method includes receiving, a computing device, video of a robotic surgical procedure; determining, by the computing device, an estimated time-to-complete ('TTC') the robotic surgical procedure based on the video; and in response to determining a deviation between the estimated TTC and an expected duration of the robotic surgical procedure exceeds a threshold, outputting an indication that the robotic surgical procedure is deviating from the expected duration.

Systems and methods for detecting delays during a surgical procedure

One example method for robotic surgery using multi-user authentication without credentials includes receiving, by a robotic surgical device, a case code associated with a medical procedure; validating the case code; determining one or more users associated with the case code; and providing access to the one or more users to the robotic surgical device to enable a robotic surgical procedure.

Robotic surgery using multi-user authentication without credentials

A system for robot-assisted surgery includes an image sensor and a display. The system further includes a controller coupled to the image sensor and the display, where the controller includes logic that when executed by the controller causes the system to perform operations. The operations may include acquiring first images of a surgical procedure with the image sensor, and analyzing the first images with the controller to identify a surgical step in the surgical procedure. The operations may further include displaying second images on the display in response to identifying the surgical step; the second images may include at least one of a diagram of human anatomy, a preoperative image, an intraoperative image, or an annotated image of one of the first images.

Step-based system for providing surgical intraoperative cues

Systems and methods for segmenting surgical videos are disclosed. One example method includes receiving, by a processor of a computing device, surgical video, the surgical video comprising at least a sequence of video frames of a surgical procedure; in response to receiving an identification of a video frame, generating, by the processor, a bookmark based on the video frame; associating, by the processor, the bookmark with the video frame; and storing, by the processor, the bookmark in a non-transitory computer-readable medium.

Systems and methods for segmenting surgical videos

One example method for identifying useful segments in surgical videos includes accessing a video of a surgical procedure and user activities of a plurality of users who have watched the video of the surgical procedure. The user activities include operations performed during playback of the video. The method further includes dividing the video into multiple segments and determining a popularity score for each of the multiple segments based on the operations. Useful segments are identified from the segments based on the popularity scores. The method further includes generating metadata for the video of the surgical procedure to include an indication of the identified useful segments and associating the metadata with the video of the surgical procedure.

Surgical video consumption by identifying useful segments in surgical videos

An apparatus, system and process for guiding a surgeon during a medical procedure to prevent surgical mistakes are described. The system may include a machine learning medical procedure server that generates one or more machine learning medical procedure models using, at least, medical procedure data captured during medical procedures performed at a plurality of different medical procedure systems. The system may also include a medical procedure system communicably coupled with the machine learning medical procedure server that receives a selected machine learning medical procedure model from the machine learning medical procedure server, and utilizes the selected machine learning medical procedure model during a corresponding medical procedure to control one or more operations of the medical procedure system.

Systems and methods for prevention of surgical mistakes

Improving surgical video consumption by identifying useful segments in surgical videos

A simulator for simulating soft body deformation includes a display system, a user interface, and a controller. The controller includes one or more processors coupled to memory that stores instructions that when executed cause the system to perform operations. The operations include generating a simulated soft body having a shape represented, at least in part, by a plurality of nodes. The operations further include determining, with at least a first machine learning model, a displacement of individual nodes included in the plurality of nodes in response to a simulated force applied to the simulated soft body. The operations further include rendering a deformation of the simulated soft body in substantially real time in response to the simulated force based, at least in part, on the displacement.

System for simulation of soft bodies

A robotic system includes a communication system coupled to communicate with one or more external devices that include first user preferences. The robotic system also includes memory storing a database including second user preferences. A controller is coupled to the user interface, the communication system, and the memory, and the controller includes logic that when executed by the controller causes the robotic system to perform operations. Operations may include retrieving the first user preferences using the communication system; retrieving the second user preferences from the database; resolving conflicts between the first user preferences and the second user preferences to create a final set of user preferences; and configuring the robotic system with the revised set of user preferences.

Growing community of inventors

Palo Alto, CA, United States of America

Martin Habbecke