Medivis, Inc.

Long Qian

Christopher Morley

Osamah J Choudhry

Osamah Choudhry

Wenbo Lan

Alfred Yu-Chih Tarng

Jesse Harrison Gallant

Danilo Gasques Rodrigues

Diego Salume

Florentin Jonas Liebmann

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to an Interaction Engine. According to various embodiments, Interaction Engine generates within a unified three-dimensional (3D) coordinate space, a virtual 3D medical model positioned according to a model pose. The Interaction Engine receives video data from a plurality of video sources. The Interaction Engine renders a first Augmented Reality (AR) display that includes concurrent display of the virtual 3D medical model and visualization of at least a portion of video data from a first video source. The Interaction Engine renders a second Augmented Reality (AR) display that includes concurrent display of the virtual 3D medical model and visualization of at least a portion of video data from a second video source.

Stereo video in augmented reality

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to a Detached Visualization Engine for detecting coordinates for a target portion of a physical anatomy in a unified three-dimensional (3D) coordinate space. The Detached Visualization Engine renders an overlay visualization of a display of medical data in alignment with the target portion of the physical anatomy. The Detached Visualization Engine applies an offset to coordinates of the overlay visualization. In some embodiments, the Detached Visualization Engine determines a secondary 3D coordinate space based on applying the offset to the entire unified 3D coordinate space. The Detached Visualization Engine renders a detached visualization according to the secondary 3D coordinate space.

Detached visualization for surgical navigation in mixed reality

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to a Registration Engine that identifies a region of interest from an infrared image portraying a target portion of a physical anatomy in a current physical pose. The Registration Engine generates a 3D point cloud of the region of interest. The Registration Engine identifies region of interest isosurface data by detecting matches between isosurface data and the 3D point cloud. The Registration Engine determines display location, at the target portion of the physical anatomy, for the region of interest isosurface data. The Registration Engine renders an Augmented Reality (AR) display of medical data with respect to the display location.

Automatic registration of landmarks for augmented reality assisted surgery

A reference marker system configured for use with a separate medical device can include a main body and a plurality of reference marker sites. The main body can have outer and inner surfaces and an overall geometry that interacts with a distinctive feature on the separate medical device. The main body can removably couple to the separate medical device along its inner surface at a specific orientation based on the distinctive feature. The reference marker sites can be coupled to the outer surface and configured to host reference markers suitable for use with a separate augmented reality system. The reference marker sites can be distributed across the outer surface at fixed positions relative to each other to form an asymmetrical fixed positional arrangement, which can be known for a medical procedure using the separate medical device without requiring any reference marker calibration based on the specific orientation.

Calibrationless reference marker system

Localizer

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to an Interaction Engine that generates within a unified three-dimensional (3D) coordinate space: (i) a 3D virtual medical model positioned according to a model pose and (ii) at least one virtual object associated with a physical instrument, the physical instrument having a current instrument pose based at least on current coordinates of one or more fiducial markers disposed on the physical instrument, in the unified 3D coordinate space. The Interaction Engine renders an Augmented Reality (AR) display that includes concurrent display of the 3D virtual medical model and the virtual object. The Interaction Engine modifies the AR display according to a virtual interaction related to the virtual object that incorporates the change of the physical instrument pose.

Virtual interaction with instruments in augmented reality

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to Registration Engine. The Registration Engine tracks positions of a fiducial marker relative to positions of an Augmented Reality (AR) headset device worn by a user. The Registration Engine receives respective registration landmarks that correspond with locations of a physical region of a patient's anatomy and the tracked positions of the fiducial marker. The Registration Engine generates an AR display of medical data at the AR headset device in alignment with the physical region of a patient's anatomy that corresponds with the respective registration landmarks.

Instrument-based registration and alignment for augmented reality environments

Various embodiments of a physical instrument are described herein. The physical instrument includes a main tubular body with a first terminal portion and a second terminal portion. The physical instrument further comprises a code platform proximate to the first terminal portion of the main tubular body. The code platform includes a plurality of different codes. The physical instrument also includes at least one of: (i) a flat tip at the second terminal portion and (ii) a passage internal to the main tubular body and extending from the first terminal portion to the second terminal portion.

Medical instrument with fiducial markers

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to Field Visualization Engine. The Field Visualization Engine tracks one or more collimator poses relative to one or more Augmented Reality (AR) headset device poses. Each respective collimator pose and each respective headset device pose corresponds to a three-dimensional (3D) unified coordinate space ('3D space'). The Field Visualization Engine generates an AR representation of a beam emanating from the collimator based at least on a current collimator pose and a current headset device pose. The Field Visualization Engine further generates an AR visualization of emanation of the beam throughout an AR display of medical data.

Visualization of predicted dosage

Various embodiments of a physical instrument are described herein. The physical instrument comprises a reference array platform having a top surface and a bottom surface. A reference array including one or more different fiducial markers is disposed on the top surface of the reference array platform. The reference array platform may have a bent physical configuration or a tilted physical configuration. An adhesive layer is disposed on the bottom surface of the reference array platform.

Adhesive fiducial markers for medical augmented reality

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to an Interaction Engine. According to various embodiments, the Interaction Engine generates, within a unified three-dimensional (3D) coordinate space: a virtual 3D medical model positioned according to a current model pose and at least one a virtual 3D hand representation. The Interaction Engine renders an AR display that includes display of the virtual 3D medical model positioned according to the current model pose and the virtual 3D hand representation. The Interaction Engine identifies selection of a slate virtual interaction based on the type of movement of the virtual 3D hand representation. The Interaction Engine modifies the AR display by rendering a virtual slate concurrently displayed with the virtual 3D medical model. The Interaction Engine adjusts the display of the virtual 3D medical model based at least on subsequent physical gestures with respect to the virtual slate.

User input and virtual touch pad in augmented reality for use in surgical settings

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to an Interaction Engine. The Interaction Engine generates, within a unified three-dimensional (3D) coordinate space: (i) a 3D virtual medical model positioned according to a model pose and (ii) at least one 3D virtual slice that corresponds with a view of respective slice layer from a plurality of slice layers associated with the 3D virtual medical model. The Interaction Engine renders an Augmented Reality (AR) display that includes concurrent display of the 3D virtual medical model and the 3D virtual slice(s). The Interaction Engine detects one or more physical gestures associated with the user and the physical instrument. The Interaction Engine identifies at least one interaction associated with the detected physical gestures and modifies the AR display according to the identified interaction.

Interactions with slices of medical data in augmented reality

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to an Interaction Engine. According to various embodiments, the Interaction Engine generates within a unified three-dimensional (3D) coordinate space: (i) a virtual 3D model container; (ii) a virtual 3D medical model positioned according to a model pose within the virtual 3D model container; and (iii) a virtual 3D representation of at least a portion of at least one of a user's hands. The Interaction Engine renders an Augmented Reality (AR) display that includes concurrent display of the virtual 3D medical model container, the virtual 3D medical model and the virtual 3D representation of the user's hands. The Interaction Engine detects one or more physical gestures associated with the user and one or more types of virtual interacts associated with a detected physical gesture(s).

Growing community of inventors

Brooklyn, NY, United States of America

Long Qian