Sony Corporation

Liangyin Yu

Ming-Chang Liu

Kuang-Man Huang

Ximin Zhang

Shigeatsu Yoshioka

Yoichi Mizutani

Masato Kajimoto

Bi Song

Quan Yuan

A method for automatic registration of landmarks in 3D and 2D images of an organ comprises: using a first set of coordinates of identified first, second and third landmarks of the organ, derived from a 3D surface representation of the organ, and a second set of coordinates of the landmarks, derived from 2D laparoscopic images of the organ, to register the three landmarks as identified in the 3D surface representation with the three landmarks as identified in the 2D images. The third landmark comprises a plurality of points defining a path between two points characterizing the first and second landmarks The identification of the first, second and third landmarks in the 3D representation and the 2D images, the derivation of the first and second sets of coordinates, and the registration, based on the derived first and second sets of coordinates, are performed automatically.

Alignment system for liver surgery

Various aspects of a system and a method to provide assistance in a surgery in presence of tissue deformation are disclosed herein. In accordance with an embodiment, the system includes an electronic device that receives one or more tissue material properties of a plurality of surface structures of an anatomical portion. One or more boundary conditions associated with the anatomical portion may also be received. Surface displacement of the anatomical portion may be determined by matching a first surface of the anatomical portion before deformation with a corresponding second surface of the anatomical portion after the deformation. The volume displacement field of the anatomical portion may be computed based on the determined surface displacement, the received one or more tissue material properties, and the received one or more boundary conditions.

System and method for providing assistance in surgery in presence of tissue deformation

An image registration system, and a method of operation of an image registration system thereof, including: an imaging unit for obtaining a pre-operation non-invasive imaging volume and for obtaining an intra-operation non-invasive imaging volume; and a processing unit including: a rigid registration module for generating a rigid registered volume based on the pre-operation non-invasive imaging volume, a region of interest module for isolating a region of interest from the intra-operation non-invasive imaging volume, a point generation module, coupled to the region of interest module, for determining feature points of the region of interest, an optimization module, coupled to the point generation module, for matching the feature points with corresponding points of the rigid registered volume for generating a matched point cloud, and an interpolation module, coupled to the optimization module, for generating a non-rigid registered volume based on the matched point cloud for display on a display interface.

Image registration system with non-rigid registration and method of operation thereof

An apparatus and method for localization of an internal organ of a subject to provide assistance during surgery, includes capture of a sequence of video frames of one or more internal organs in a body of a subject by insertion of an image capture device in a body of the subject. The apparatus further includes circuitry that computes appearance data of an internal organ of interest in a test video frame selected from the captured sequence of video frames. Orientation data of the internal organ of interest is determined within the body of the subject based on a known orientation of the image capture device and a maximum correlation of the test video frame with a specified anatomical template for the internal organ of interest. Supplemental information is generated to localize at least a portion of the internal organ of interest of the subject in the surgery.

Internal organ localization of a subject for providing assistance during surgery

A surgical assistive apparatus and method to provide assistance during surgery, includes an organ boundary localization circuit that selects a test video frame from a captured sequence of video frames and derives a first global region boundary of an internal organ of interest in the selected test video frame by integration of an appearance likelihood result, a global segmentation result, and a global edge detection result associated with the internal organ of interest. A plurality of local region boundaries are determined for a plurality of local sub regions of the internal organ of interest and a second global region boundary is generated for the internal organ of interest based on the determined plurality of local region boundaries and the first global region boundary. The internal organ of interest is localized based on the generated second global region boundary.

Boundary localization of an internal organ of a subject for providing assistance during surgery

A surgical assistive device, which handles co-localization of an internal organ of interest and a neighboring organ based on images obtained during surgery, selects a test video frame from a sequence of video frames and generates first localization likelihood values and second localization likelihood values for the internal organ of interest and the neighboring organ, respectively. The surgical assistive device determines extent of anatomical adjacency boundary between the internal organ of interest and the neighboring organ, based on the first localization likelihood values and the second localization likelihood values. The surgical assistive device segments the test video frame into a plurality of regions and generates a first region based on integration of a first set of regions from the plurality of regions. The surgical assistive device further localizes the internal organ of interest based on first region, in accordance with the extent of anatomical adjacency boundary.

Co-localization of multiple internal organs based on images obtained during surgery

Various aspects of a surgical assistive device and method to provide assistance in neurosurgery are disclosed herein. The surgical assistive device includes circuitry that determines initial positional shift in the anatomical portion of a subject caused by an incision into a protective portion. A first depth value is estimated based on the plurality of stereo images that are captured from different specified positions of the surgical image capture device to capture views of the anatomical surface of the anatomical portion through the incision. A residual shift in a position of a surgical region of interest is determined with respect to the estimated first depth value. Surgical guidance information is generated to indicate a correct depth of the surgical region of interest in a surgical cavity, based on the determined residual shift. Display of the generated surgical guidance information is controlled on a display screen coupled to the surgical assistive device.

Surgical assistive device and method for providing assistance in surgery of anatomical portions of internal organ affected by intraoperative shift

Various aspects of an image-processing system and method to reconstruct a three-dimensional (3D) anatomical surface of an anatomical portion are disclosed herein. The system includes an image-processing device configured to receive a plurality of stereo images of the anatomical portion. A first set of key points with a point density greater than a threshold value is identified. A second set of key points is determined based on filtration of one or more outlier key points from the identified first set of key points. A 3D anatomical surface of the anatomical portion is reconstructed based on disparity of one or more matched key points in the determined second set of key points and a smoothing operation performed on the disparity.

Image processing system and method to reconstruct a three-dimensional (3D) anatomical surface

Various aspects of a system and a method to process multimodal images are disclosed herein. In accordance with an embodiment, the system includes an image-processing device that generates a structured point cloud, which represents edge points of an anatomical portion. The structured point cloud is generated based on shrink-wrapping of an unstructured point cloud to a boundary of the anatomical portion. Diffusion filtering is performed to dilate edge points that correspond to the structured point cloud to mutually connect the edge points on the structured point cloud. A mask is created for the anatomical portion based on the diffusion filtering.

System and method for processing multimodal images

A gesture recognition system using a skin-color based method combined with motion information to achieve real-time segmentation. A Kalman filter is used to track the centroid of the hand. The palm center, palm bottom, as well as the largest distance from the palm center to the contour from extracted hand mask are computed. The computed distance to a threshold is then compared to decide if the current posture is 'open' or 'closed.' In a preferred embodiment, the transition between the &#x201c;open&#x201d; and &#x201c;closed&#x201d; posture to decide if the current gesture is in &#x201c;select&#x201d; or &#x201c;grab&#x201d; state.

Gesture recognition system for TV control

A system for performing an image categorization procedure includes an image manager with a keypoint generator, a support region filter, an orientation filter, and a matching module. The keypoint generator computes initial descriptors for keypoints in a test image. The support region filter and the orientation filter perform respective filtering procedures upon the initial descriptors to produce filtered descriptors. The matching module compares the filtered descriptors to one or more database image sets for categorizing said test image. A processor of an electronic device typically controls the image manager to effectively perform the image categorization procedure.

System and method for effectively performing an image categorization procedure

Certain aspects of an apparatus and method for method and apparatus for tissue region identification may include segmenting the image into a plurality of regions, filtering out regions in the plurality of regions which are curvilinear, and isolating a target area where the tissue sample is identified as the plurality of regions not filtered.

Method and apparatus for tissue region identification

A system and method for effectively performing an integrated segmentation procedure comprises an image segmenter that includes a texture modeler, a contrast modeler, and a model integrator. The texture modeler creates a texture model based upon an original image. Similarly, the contrast modeler creates a contrast model based upon the original image. The model integrator then performs a model integration procedure to create a final segmented image by integrating the texture model and the contrast model according to a calculated texture model metric. A processor of an electronic device typically controls the image segmenter to perform the integrated segmentation procedure.

System and method for effectively performing an integrated segmentation procedure

A system for performing a scene representation procedure includes an image manager that processes source images from a given scene to define subscenes in the source images. The image manager creates an image understanding graph for each of the source images, and also creates a scene representation graph for each of the source images based upon the corresponding subscenes and certain image characteristics. The image manager further generates an integrated scene representation to represent all of the source images with a single representation. A processor of an electronic device controls the image manager to perform the scene representation procedure.

System and method for effectively performing a scene representation procedure

Certain aspects of an apparatus and method for orienting tissue samples for comparison may include incrementally rotating orientation of a first image by a predetermined rotation angle while maintaining orientation of a second image at a fixed angle, checking alignment of the orientation of the first image with the orientation of the second image at each predetermined rotation angle by matching a plurality of points in the first image and the second image, determining whether a predetermined rotation angle is a correct rotation angle for alignment based on a count of the plurality of points being greater than a threshold value and rotating to the next predetermined rotation angle when the count of the plurality of points is less than or equal to a threshold value.

Growing community of inventors

Fremont, CA, United States of America

Liangyin Yu