Verb Surgical Inc.

Jocelyn Elaine Barker

Pablo E Garcia Kilroy

Varun Kejriwal Goel

Richard William Timm

Meysam Torabi

Mona Fathollahi Ghezelghieh

Darrick Tyler Sturgeon

Taylor W Aronhalt

Danyal Fer

Peijmon Kasravi

Amer Ghanem

Nishant Shailesh Sahni

Rami Abukhalil

An annotation system facilitates collection of labels for images, video, or other content items relevant to training machine learning models associated with surgical applications or other medical applications. The annotation system enables an administrator to configure annotation jobs associated with training a machine learning model. The job configuration controls presentation of content items to various participating annotators via an annotation application and collection of the labels via a user interface of the annotation application. The annotation application enables the participating annotators to provide inputs in a simple and efficient manner, such as by providing gesture-based inputs or selecting graphical elements associated with different possible labels.

Annotation system for surgical content

In one aspect, the process receives a current frame of the endoscope video at a current time of the live surgical session, wherein the current time is among a sequence of prediction time points for making continuous RSD predictions during the live surgical session. The process next randomly samples additional frames of the endoscope video corresponding to the elapsed portion of the live surgical session. The process then combines the sampled frames and the current frame in the temporal order to obtain a set of N frames. Next, the process feeds the set of N frames into a trained model for the given surgical procedure. The process subsequently outputs a current RSD prediction based on the set of N frames. Other aspects are also described and claimed.

Deep-learning-based real-time remaining surgery duration (rsd) estimation

Embodiments described herein include a process for detecting energy tool activations. The process can begin by receiving a surgical video of a surgical procedure involving energy tool activations. The process then applies a sequence of sampling windows to the surgical video to generate a sequence of windowed samples of the surgical video. Next, for each windowed sample in the sequence of windowed samples, the process applies a deep-learning model to a sequence of video frames within the windowed sample to generate an activation/non-activation inference and a confidence level associated with the activation/non-activation inference for the windowed sample. As a result, a sequence of activation/non-activation inferences and a sequence of associated confidence levels are generated. The process subsequently identifies a sequence of activation events in the surgical video based on the sequence of activation/non-activation inferences and the sequence of associated confidence levels.

Energy tool activation detection in surgical videos using deep learning

Embodiments described herein provide various techniques and systems for building machine-learning surgical tool presence/absence detection models for processing surgical videos and predicting whether a surgical tool is present or absent in each video frame of a surgical video. In one aspect, a process for ensuring patient safety during a laparoscopic or robotic surgery involving an energy tool is disclosed. The process can begin receiving a real-time control signal indicating an operating state of an energy tool during the surgery. Next, the process receives real-time endoscope video images of the surgery. The process simultaneously applies a machine-learning surgical tool presence/absence detection model to the real-time endoscope video images to generate real-time decisions on a location of the energy tool in the real-time endoscope video images. The process then checks the real-time control signal against the real-time decisions to identify an unsafe event and takes a proper action when an unsafe event is identified.

Real-time surgical tool presence/absence detection in surgical videos

An analysis system trains a machine learning model to detect stapling events from a video of a surgical procedure. The machine learning model detects times when stapling events occur as well as one or more characteristics of each stapling event such as length of staples, clamping time, or other characteristics. The machine learning model is trained on videos of surgical procedures identifying when stapling events occurred through a learning process. The machine learning model may be applied to an input video to detect a sequence of stapler events. Stapler event sequences may furthermore be analyzed and/or aggregated to generate various analytical data relating to the surgical procedures for applications such as inventor management, performance evaluation, or predicting patient outcomes.

Video-based analysis of stapling events during a surgical procedure using machine learning

Embodiments described herein provide a surgical duration estimation system for continuously predicting real-time remaining surgical duration (RSD) of a live surgical session of a given surgical procedure based on a real-time endoscope video of the live surgical session. In one aspect, the process receives a current frame of the endoscope video at a current time of the live surgical session, wherein the current time is among a sequence of prediction time points for making continuous RSD predictions during the live surgical session. The process next randomly samples N−1 additional frames of the endoscope video corresponding to the elapsed portion of the live surgical session between the beginning of the endoscope video corresponding to the beginning of the live surgical session and the current frame corresponding to the current time. The process then combines the N−1 randomly sampled frames and the current frame in the temporal order to obtain a set of N frames. Next, the process feeds the set of N frames into a trained model for the given surgical procedure. The process subsequently outputs a current RSD prediction based on the set of N frames.

Growing community of inventors

San Jose, CA, United States of America

Jocelyn Elaine Barker