Capsovision Inc.

Seiko Epson Corporation

Intuitive Surgical Operations, Inc.

Wu, Chenyu

Wang, Kang-huai

Chenyu Wu

Kang-Huai Wang

Jing Xiao

Gordon Cook Wilson

Yi Xu

Tao Zhao

Wenyi Zhao

Brian David Hoffman

Mark Alfred Hadley

David Hirvonen

Simon Di Maio

Catherine J Mohr

Christopher J Hasser

Brian Edward Miller

Myrian J Curet

Systems and methods are provided for processing images captured by a capsule camera system. According to one method, a frame sequence comprises panoramic picture representative of a GI scene covering substantially a lateral 360-degree Field-of-View is received. CADe (computer-assisted detection) is applied to detect one or more target abnormalities in the frame sequence, where said applying the computer-assisted detection (CADe) results in one or more CADe frames and each CADe frame contains at least one detection area containing a detected abnormality or a false positive. Circle shift is applied laterally to a target CADe frame to shift a detection area to a lateral center or close to the lateral center of the target CADe frame. The CADe frames comprising the target CADe frame are provided with the detection area shifted to the lateral center or close to the lateral center of the target CADe frame.

System and method for reviewing capsule images with detected regions of interest

A method and apparatus of objective assessment of images captured from a human gastrointestinal (GI) tract are disclosed. According to this method, one or more images captured using an endoscope when the endoscope is inside the human gastrointestinal (GI) tract are received. Whether there is any specific target object is checked. When one or more specific target objects in the images are detected: areas of the specific target objects in the images are determined; an objective assessment score is derived based on the areas of the specific target objects in a substantial number of images from the images; where the step of detecting the specific target objects is performed using an artificial intelligence process.

Method and apparatus for objective assessment of gastrointestinal conditions based on images captured in the GI tract

A method and apparatus for sharpening gastrointestinal (GI) images are disclosed. A target distance between the target region and the imaging apparatus is determined for a target region in the regular image. One or more filter parameters of a de-blurring filter are selected from stored filter parameters according to the target distance. A processed target region is generated by applying the de-blurring filter to the target region to improve sharpness of the target region. A method for characterizing an imaging apparatus is also disclosed. The imaging apparatus is placed under a controlled environment. Test pictures for one or more test patterns are captured at multiple test distances in a range including a focus distance using the imaging apparatus. One or more parameters associated a target point spread function are determined from each test picture for characterizing image formation of the imaging apparatus at the selected distance.

Method and apparatus of sharpening of gastrointestinal images based on depth information

A method of processing images captured using an endoscope comprising a camera is disclosed. According to this method, regular images captured by the camera are received while the endoscope travels through a human gastrointestinal (GI) tract. The regular images are mosaicked to determine any missed or insufficiently imaged area in a section of the human GI tract already travelled by the endoscope. If any missed or insufficiently imaged area is detected, information regarding any missed or insufficiently imaged area is provided. When a target area in the regular images is lack of parallax, the target area is determined as one missed area and an edge corresponding to a structure of the human lumen is highlighted. For a capsule endoscope, the endoscope can be configured to be controlled or steered to move so as to re-image the missed or insufficiently imaged area.

Method and apparatus for detecting missed areas during endoscopy

A method and apparatus of image stitching using confidence level of image matching on image pair to be stitched are disclosed. According to the present invention, the first quality of image matching is determined for a current image based on feature matching on first image pairs corresponding to the current image and a neighboring image set. If the first quality of image matching for at least one first image pair satisfies a first quality criterion, then the second quality of image matching is determined based on pixel-domain matching for one or more candidate image pairs, where each candidate image pair has a corresponding first quality of image matching satisfying the first quality criterion. If the second quality of image matching for a candidate image pair satisfies a second quality criterion, then the matched image pair is stitched.

Reconstruction of images from an in vivo multi-camera capsule with two-stage confidence matching

An integrated image sensor for capturing a mixed structured-light image and regular image using an integrated image sensor are disclosed. The integrated image sensor comprises a pixel array, one or more output circuits, one or more analog-to-digital converters, and one or more timing and control circuits. The timing and control circuits are arranged to perform a set of actions including capturing a regular image and a structured-light image. According to the present invention, the structured-light image captured before or after the regular image is used to derive depth or shape information for the regular image. An endoscope based on the above integrated image sensor is also disclosed. The endoscope may comprises a capsule housing adapted to be swallowed, where the components of integrated image sensor, a structured light source and anon-structured light source are enclosed and sealed in the capsule housing.

Single image sensor for capturing mixed structured-light images and regular images

A method and apparatus of processing images captured from human gastrointestinal (GI) tract by a capsule camera are disclosed. High frame-rate images captured from human gastrointestinal (GI) tract by a capsule camera are received for processing. The high frame-rate images comprise first images at a first spatial resolution corresponding to a regular frame rate and second images at a second spatial resolution, the first images and the second images are interleaved, and the second spatial resolution is lower than the first spatial resolution. Motion models among the high frame-rate images are derived by applying image registration to the high frame-rate images. The high frame-rate images are stitched according to the motion models to generate stitching outputs comprising stitched images and non-stitched images. The stitching outputs are provided.

Method and apparatus for image stitching of images captured using a capsule camera

A method for sharpening gastrointestinal (GI) images and an apparatus thereof are disclosed. A first target distance between a first target region in a regular image and an imaging apparatus is determined, and a second target distance between a second target region in the regular image and the imaging apparatus is determined. One or more first filter parameters for a first de-blurring filter and one or more second filter parameters for a second de-blurring filter are selected from stored filter parameters according to the first and second target distances respectively. A first processed target region is generated by applying the first de-blurring filter to the first target region to improve sharpness of the first target region. A second processed target region is generated by applying the second de-blurring filter to the second target region to improve sharpness of the second target region.

A method and apparatus for imaging a body lumen are disclosed. According to the method, an imaging apparatus is induced into the body lumen. Structured light from the imaging apparatus is projected into the body lumen. The structured light reflected from anatomical features in the body lumen is detected by the imaging apparatus. A first structured light image is generated from the detected structured light by the imaging apparatus. Non-structured light is emitted from the imaging apparatus into the body lumen. The non-structured light reflected from the anatomical features in the body lumen is detected by the imaging apparatus. A non-structured light image is generated from the detected non-structured light by the imaging apparatus. The frame period of the first structured light image is shorter than the frame period of the non-structured light image. In one embodiment, the imaging apparatus corresponds to a capsule endoscope.

A method and apparatus for estimating or measuring a physical area or physical volume of an object of interest in one or more images captured using an endoscope are disclosed. According to the present method, one or more structured-light images and one or more regular images captured using an imaging apparatus are received. An object of interest in the regular images is determined. Distance information associated with the object of interest with respect to the imaging apparatus is derived from the structured-light images. The physical area size or physical volume size of the object of interest is determined based on the regular images and the distance information. The imaging apparatus can be a capsule endoscope or an insertion endoscope.

Method and apparatus for estimating area or volume of object of interest from gastrointestinal images

A method and device for improving the accuracy of depth information derived from a structured-light image for a regular image are disclosed. In one example, an additional structured-light image is captured before a first structured-light image or after a regular image. The depth information for the regular image can be derived from the first structured-light image and corrected by incorporating depth information from the additional structured-light image. A model for depth information can be used to predict or interpolate depth information for the regular image. In another example, two regular sub-images may be captured with a structured-light image in between. If substantial frame differences or substantial global motion vector/block motion vectors are detected, the two regular sub-images will not be combined in order to avoid possible motion smear. Instead, one of the two sub-images will be selected and scaled as the output regular image.

A method and apparatus for estimating or measuring a physical area or physical volume of an object of interest in one or more images captured using an endoscope are disclosed. According to the present method, an object of interest in an image or images is determined. Also, the endoscope captures one or more structured-light images to derive distance information associated with the object of interest with respect to the camera of the endoscope. The physical area size or physical volume size of the object of interest is then determined based on the image or images, and the distance information.

An integrated image sensor for capturing a mixed structured-light image and regular image using an integrated image sensor are disclosed. The integrated image sensor comprises a pixel array, one or more output circuits, one or more analog-to-digital converters, and one or more timing and control circuits. The timing and control circuits are arranged to perform a set of actions including capturing a regular image and a structured-light image. According to the present invention, the structured-light image captured before or after the regular image is used to derive depth or shape information for the regular image. An endoscope based on the above integrated image sensor is also disclosed. The endoscope may comprises a capsule housing adapted to be swallowed, where the components of integrated image sensor, a structured light source and a non-structured light source are enclosed and sealed in the capsule housing.

A method and apparatus for capturing images of a scene using a capsule device including a camera are disclosed. An image sequence is captured using the camera when the capsule device travels through a human gastrointestinal tract. Also, structured-light images are captured using the camera by projecting structured light to one or more objects in a field of view of the camera when the capsule device travels through the human gastrointestinal tract. The structured-light images are interleaved with regular images in the image sequence. The distance information with respect to the capsule camera associated with objects of the selected image is derived. Both the image sequence and the distance information are outputted. A method of determining the size of an object of interest utilizing the distance information is also disclosed. In another method, the distance information is used to scale object or adjust intensities.

Growing community of inventors

Sunnyvale, CA, United States of America

Chenyu Wu