Shanghai United Imaging Intelligence Co., Ltd.

Meng Zheng

Ziyan Wu

Abhishek Sharma

Srikrishna Karanam

Benjamin Planche

Terrence Chen

Arun Innanje

Zhongpai Gao

Ren Li

Shanhui Sun

Liangchen Song

Elena Zhao

Zikui Cai

Pierre Sibut-Bourde

A person's privacy is protected by the law in many settings and disclosed herein are systems, methods, and instrumentalities associated with anonymizing an image of a person while still preserving the visual saliency and/or utility of the image for one or more downstream tasks. These objectives may be accomplished using various machine-learning (ML) techniques such as ML models trained for extracting identifying and residual features from the input image as well as ML models trained for transforming the identifying features into identity-concealing features and for preserving the utility features of the image. An output image may be generated based on the various ML models, wherein the identity of the person may be substantially disguised in the output image while the background and utility attributes of the original image may be substantially maintained in the output image.

Systems and methods for anonymizing images

Described herein are systems, methods, and instrumentalities associated with estimating the motions of multiple 3D points in a scene and predicting a view of scene based on the estimated motions. The tasks may be accomplished using one or more machine-learning (ML) models. A first ML model may be used to predict motion-embedding features for a temporal state of a scene, based on motion-embedding features for previous states. A second ML model may be used to predict a motion field representing displacement or deformation of the multiple 3D points from a source time to a target time. Then, a third ML model may be used to predict respective image properties of the 3D points based on their updated locations at the target time and/or a viewing direction. An image of the scene at the target time may then be generated based on the predicted image properties of the 3D points.

Systems and methods for motion estimation and view prediction

Described herein are systems, methods, and instrumentalities associated with automatically annotating a 3D image dataset. The 3D automatic annotation may be accomplished based on a 2D annotation provided by an annotator and by propagating the 2D annotation through multiple images of a sequence of 2D images associated with the 3D image dataset. The automatically annotated 3D image dataset may then be used to annotate other 3D image datasets based on similarities between the first 3D image dataset and the other 3D image datasets. The automatic annotation of the first 3D image dataset and/or the other 3D image datasets may be conducted based on one or more machine-learning models trained for performing those tasks.

Systems and methods for annotating 3D data

The physical characteristics of one or more anatomical structures of a person may change in accordance with conditions surrounding the determination of such physical characteristics. Machine learning based techniques may be used to determine a template representation of the one or more anatomical structures that may indicate the physical characteristics of the one or more anatomical structures free of the impact imposed by changing conditions. The template representation may then be used to predict the physical characteristics of the one or more anatomical structures under a new set of conditions, without subjecting the person to additional medical scans.

Systems and methods for determining anatomical deformations

The 3D pose of a person may be estimated by triangulating 2D representations of body keypoints (e.g., joint locations) of the person. The triangulation may leverage various metrics such as confidence scores associated with the 2D representations of a keypoint and/or temporal consistency between multiple 3D representations of the keypoint. The 2D representations may be arranged into groups, a candidate 3D representation may be determined for each group, taking into account of the confidence score of each 2D representation in the group, and the candidate 3D representation that has the smallest error may be used to represent the keypoint. Other 3D representation(s) of the keypoint determined from images taken at different times may be used to refine the 3D representation of the keypoint.

Systems and methods for determining 3D human pose based on 2D keypoints

A method and a system for managing healthcare records of a user are provided. The method includes storing an electronic medical record related to the user in form of a non-fungible token (NFT) written to a blockchain, associating a smart contract to the NFT in the blockchain, authorizing a request to access the electronic medical record related to the user based on the defined ownership of the electronic medical record stored in the blockchain, identifying one or more NFTs from the blockchain comprising one or more electronic medical records related to the user based on processing of the identifier information in associated one or more smart contracts therewith, in response to the request, and sending the one or more electronic medical records corresponding to the identified one or more NFTs to a requestor associated with the request.

Patient care record management system

An automated process for data annotation of medical images includes obtaining image data from an imaging sensor, partitioning the image data, identifying an object of interest in the partitioned image data, generating an initial contour with one or more control points with respect to the object of interest, identifying a manual adjustment of one of the control points, automatically adjust a position of at least one other control point within a predetermined range of the manually adjusted control point to a new position, the new position of the at least one other control point and manually adjusted control point defining a new contour, and generating an updated image with the new contour and corresponding control points.

Interactive contour refinements for data annotation

An apparatus for automated collision avoidance includes a sensor configured to detect an object of interest, predicting a representation of the object of interest at a future point in time, calculating an indication of a possibility of a collision with the object of interest based on the representation of the object of interest at the future point in time, and executing a collision avoidance action based on the indication.

Automated collision avoidance in medical environments

A human model such as a 3D human mesh may be generated for a person in a medical environment based on one or more images of the person. The images may be captured using a sensing device that may be attached to an existing medical device such as a medical scanner in the medical environment. Such an arrangement may ensure that unblocked views of the person (e.g., body keypoints of the person) may be obtained and used to generate the human model. The position of the medical device in the medical environment may be determined and used to facilitate the human model construction such that the pose and body shape of the person in the medical environment may be accurately represented by the human model.

Systems and methods for human model recovery

Human model recovery may be realized utilizing pre-trained artificially neural networks. A first neural network may be trained to determine body keypoints of a person based on image(s) of a person. A second neural network may be trained to predict pose parameters associated with the person based on the body keypoints. A third neural network may be trained to predict shape parameters associated with the person based on depth image(s) of the person. A 3D human model may then be generated based on the pose and shape parameters respectively predicted by the second and third neural networks. The training of the second neural network may be conducted using synthetically generated body keypoints and the training of the third neural network may be conducted using normal maps. The pose and shape parameters predicted by the second and third neural networks may be further optimized through an iterative optimization process.

Human model recovery using deep learning techniques

The shape and/or location of an organ may change in accordance with changes in the body shape and/or pose of a patient. Described herein are systems, methods, and instrumentalities for automatically determining, using an artificial neural network (ANN), the shape and/or location of the organ based on human models that reflect the body shape and/or pose the patient. The ANN may be trained to learn the spatial relationship between the organ and the body shape or pose of the patient. Then, at an inference time, the ANN may be used to determine the relationship based on a first patient model and a first representation (e.g., a point cloud) of the organ so that given a second patient model thereafter, the ANN may automatically determine the shape and/or location of the organ corresponding to the body shape or pose of the patient indicated by the second patient model.

Automatic organ geometry determination

Automatically validating the calibration of an visual sensor network includes acquiring image data from visual sensors that have partially overlapping fields of view, extracting a representation of an environment in which the visual sensors are disposed, calculating one or more geometric relationships between the visual sensors, comparing the calculated one or more geometric relationships with previously obtained calibration information of the visual sensors, and verifying a current calibration of the visual sensors based on the comparison.

Automated calibration and validation of sensor network

Described herein are systems, methods, and instrumentalities associated with generating a multi-dimensional representation of a medical environment based on images of the medical environments. Various pre-processing and/or post-processing operations may be performed to supplement and/or improve the multi-dimensional representation. These operations may include determining semantic information associated with the medical environment based on the images and adding the semantic information to the multi-dimensional representation in addition to space and time information. The operations may also include anonymizing a person presented in the multi-dimensional representation, adding synthetic views to the multi-dimensional representation, improving the quality of the multi-dimensional representation, etc. The multi-dimensional representation of the medical environment generated using these techniques may allow a user to experience and explore the medical environment, for example, via a virtual reality device.

Systems and methods for recording medical environments

Systems, methods, and instrumentalities are described herein for constructing a multi-view patient model (e.g., a 3D human mesh model) based on multiple single-view models of the patient. Each of the single-view models may be generated based on images captured by a sensing device and, dependent on the field of the view of the sensing device, may depict some keypoints of the patient's body with a higher accuracy and other keypoints of the patient's body with a lower accuracy. The multi-view patient model may be constructed using respective portions of the single-view models that correspond to accurately depicted keypoints. This way, a comprehensive and accurate depiction of the patient's body shape and pose may be obtained via the multi-view model even if some keypoints of the patient's body are blocked from a specific sensing device.

Multi-view patient model construction

Systems, methods and instrumentalities are described herein for automating a medical environment. The automation may be realized using one or more sensing devices and at least one processing device. The sensing devices may be configured to capture images of the medical environment and provide the images to the processing device. The processing device may determine characteristics of the medical environment based on the images and automate one or more aspects of the operations in the medical environment. These characteristics may include, e.g., people and/or objects present in the images and respective locations of the people and/or objects in the medical environment. The operations that may be automated may include, e.g., maneuvering and/or positioning a medical device based on the location of a patient, determining and/or adjusting the parameters of a medical device, managing a workflow, providing instructions and/or alerts to a patient or a physician, etc.

Growing community of inventors

Troy, NY, United States of America

Meng Zheng