Verily Life Sciences LLC

Palo Alto Networks, Inc.

Xing Jin

Martin Habbecke

Joëlle K Barral

Lin Yang

Joëlle Barral

Caitlin Q Donhowe

Zhaoyan Xu

Xin Ouyang

Tongbo Luo

Gianni Campion

Yanhui Jia

Systems and methods for eye tracking and content tagging in minimally invasive surgical videos are described herein. Minimally invasive surgical videos may be captured while performing robotic surgeries. Robotic surgical systems described herein include robotic arms with interchangeable surgical tools. An endoscope at the end of one of the robotic arms captures video of the surgical procedure. The video is displayed on a display of the surgical system and an eye tracking device captures data corresponding to a gaze direction of the user on the display. Content tags are automatically generated in the image data based on areas of focus of the user. Information within the content tag may be generated based on surgical procedure steps derived from the image data, a log of current surgical procedures, and image data of previous surgical procedures.

Automatic content tagging in videos of minimally invasive surgeries

A system includes a display, and a database including surgical videos, images of organs in a human body obtained from a medical imaging device, and images of disease in a human body obtained from the medical imaging device. A controller including a processor is coupled to memory, the database, and the display, and the memory stores information that when executed by the processor causes the system to perform operations. For example, the processor may determine first organ information from the images of the organs, and first disease information from the images of the disease. The processor my calculate a similarity score between the first organ information and the first disease information and second disease information and second organ information indexed to the surgical videos. The processor selects one or more of the surgical videos based on the similarity score, and displays the surgical videos on the display.

Surgical video retrieval based on preoperative images

One example method for improving robotic surgical safety via video processing includes identifying, during a robotic surgical procedure, one or more surgical tools using one or more images of the surgical procedure captured by a camera, the robotic surgical procedure employing a robotic surgical device controlling the one or more identified surgical tools; predicting, for at least one of the one or more images, one or more loaded surgical tools that are controlled by the robotic surgical device and should be in a field of view of the camera; comparing the one or more identified surgical tools with the one or more loaded surgical tools; determining that at least one of the one or more loaded surgical tools does not match any of the one or more identified surgical tools; and causing the at least one loaded tool of the robotic surgical device to be disabled.

Robotic surgical safety via video processing

A surgical simulator for simulating a surgical scenario comprises a display system, a user interface, and a controller. The controller includes one or more processors coupled to memory that stores instructions that when executed cause the system to perform operations. The operations include generating simulated surgical videos, each representative of the surgical scenario. The operations further include associating simulated ground truth data from the simulation with the simulated surgical videos. The ground truth data corresponds to context information of at least one of a simulated surgical instrument, a simulated anatomical region, a simulated surgical task, or a simulated action. The operations further include annotating features of the simulated surgical videos based, at least in part, on the simulated ground truth data for training a machine learning model.

Surgical simulator providing labeled data

Systems and methods for depth detection and virtual modeling using surgical robots during a surgical procedure are described. The robotic surgical systems include robotic arms with interchangeable surgical tools. An endoscope at the end of one of the robotic arms includes a depth sensor for detecting a distance from the camera to patient anatomy. The depth data and image data are used to generate a feature model of the patient anatomy for reference by a surgeon. The feature model is displayed to the surgeon in an augmented reality view with preoperative and intraoperative images mapped thereon for reference and guidance of a surgical procedure.

Robotic surgery depth detection and modeling

Systems and methods for identifying surgery tools of a robotic surgery system in surgery video based on robot kinematics are described. Kinematic data describing a position of a first robotic surgery arm including a surgery tool is used to determine a nominal position of the first robotic surgery arm. A search area within image data obtained during a robotic surgery from an endoscope attached to a second robotic surgery arm is determined based on the kinematic data and a nominal position of the surgery tool. The surgery tool is identified within the search area of the image data using an object detection technique.

Surgery tool segmentation with robot kinematics

Examples of systems and methods for detecting delays during a surgical procedure are disclosed. For example, one disclosed method includes receiving, a computing device, video of a robotic surgical procedure; determining, by the computing device, an estimated time-to-complete ('TTC') the robotic surgical procedure based on the video; and in response to determining a deviation between the estimated TTC and an expected duration of the robotic surgical procedure exceeds a threshold, outputting an indication that the robotic surgical procedure is deviating from the expected duration.

Systems and methods for detecting delays during a surgical procedure

One example method for detecting phases of a surgical procedure via video processing includes accessing a video of the surgical procedure and dividing the video into one or more blocks, each of the blocks containing one or more video frames. For each of the blocks, the method includes applying a prediction model on the video frames of the respective block to obtain a phase prediction for each of the video frames. The prediction model is configured to predict, for an input video frame, one of the plurality of phases of the surgical procedure. The method further includes generating an aggregated phase prediction for the respective block by aggregating the phase predictions of the video frames, and modifying the video of the surgical procedure to include an indication of a predicted phase of the respective block based on the aggregated phase prediction.

Surgical workflow and activity detection based on surgical videos

A system for robot-assisted surgery includes an image sensor and a display. The system further includes a controller coupled to the image sensor and the display, where the controller includes logic that when executed by the controller causes the system to perform operations. The operations may include acquiring first images of a surgical procedure with the image sensor, and analyzing the first images with the controller to identify a surgical step in the surgical procedure. The operations may further include displaying second images on the display in response to identifying the surgical step; the second images may include at least one of a diagram of human anatomy, a preoperative image, an intraoperative image, or an annotated image of one of the first images.

Step-based system for providing surgical intraoperative cues

The use of browser context in detecting malware is disclosed. A client device requests content from a remote server. Data received by the client device from the remote server is transmitted to an external scanner for analysis by the external scanner. The external scanner is configured to use a browser executed in an instrumented virtual machine environment to analyze the data provided by the client device. The client device is configured to request the content from the remote server using a browser extension configured to retrieve data and provide the retrieved data to the external scanner without rendering the retrieved data.

Using browser context in evasive web-based malware detection

One example method for identifying useful segments in surgical videos includes accessing a video of a surgical procedure and user activities of a plurality of users who have watched the video of the surgical procedure. The user activities include operations performed during playback of the video. The method further includes dividing the video into multiple segments and determining a popularity score for each of the multiple segments based on the operations. Useful segments are identified from the segments based on the popularity scores. The method further includes generating metadata for the video of the surgical procedure to include an indication of the identified useful segments and associating the metadata with the video of the surgical procedure.

Surgical video consumption by identifying useful segments in surgical videos

Techniques for providing an intelligent-interaction honeypot for IoT devices in accordance with some embodiments. In some embodiments, a system/process/computer program product for providing an intelligent-interaction honeypot for IoT devices includes receiving a request from an attacker sent to an IP address that is associated with a honeypot instance for Internet of Things (IoT) devices; determining a response to the request using a data store that stores a plurality of responses and associated IoT device information, wherein the plurality of responses and associated IoT device information is generated based on automated machine learning of active probing of physical IoT devices on the Internet; and sending the response from the honeypot instance for IoT devices to the attacker, wherein the attacker is unable to detect that the response is associated with an emulated IoT device.

Growing community of inventors

San Jose, CA, United States of America

Xing Jin