3d cameras or sensors inputting to multi-modal generative artificial intelligence models trained on images or videos

Abstract

3D cameras may serve as an input to a multi-modal generative artificial intelligence (GAI) model operating on a processor coupled to a non-transitory computer readable medium. Examples of 3D cameras include direct or indirect time-of-flight sensors or structured light systems and may also be coupled to 2D cameras. The GAI comprises a vision transformer configured to analyze an item in an input video or image. The vision transformer comprises self-attention and positional encoding layers. The GAI may be trained using reinforcement learning or fine-tuning involving training images or videos, and it may perform data fusion by combining the 3D information with data from other sensors. The GAI may perform anomalous occurrence detection by training on images or videos corresponding to normal occurrences. The GAI detects differences in an image or video that fall outside of a threshold value. The GAI may also provide safeguards for privacy issues.