Pelco, Inc.

Nvidia Corporation

Farzin Aghdasi

Greg Max Millar

Lei Wang

Parthasarathy Sriram

Hongwei Zhu

Chien-Min Huang

Milind R Naphade

Fnu Ratnesh Kumar

Sujit Biswas

Shu Yang

Stephen J Mitchell

Subhashree Radhakrishnan

Arman Toorians

Anil Ubale

Yan Zhai

Aaron Bartholomew

Wei Su

Bhanu Nagendra Pisupati

Vinay Kolar

Zhan Jiang Ma

Ratnesh Kumar

Kirsten A Medhurst

Bryan K Neff

Lawrence E Bakst

James G Millar

Edwin Weill

Praveen Mehrotra

Partha Sriram

Zhiding Yu

Bryan R Grziwok

Shuo Wang

Seunghwan Cha

Zeyu Zhao

Shangru Li

Evan McLaughlin

Arihant Jain

Sezai Sablak

Jennifer Joyner

Systems and methods disclosed relate to generating training data. In one embodiment, the disclosure relates to systems and methods for generating training data to train a neural network to detect and classify objects. A simulator obtains 3D models of objects, and simulates 3D environments comprising the objects using physics-based simulations. The simulations may include applying real-world physical conditions, such as gravity, friction, and the like on the objects. The system may generate images of the simulations, and use the images to train a neural network to detect and classify the objects from images.

Object simulation using real-world environments

Calibration of various sensors may be difficult without specialized software to process intrinsic and extrinsic information about the sensors. Certain types of input files, such as image files, may also lack certain information, like depth information, to effectively translate regions of interest between images taken from a different perspective. Landmarks can be used to establish points for associating regions of interest between images taken from a different perspective and provided as an overlay to verify sensor calibration.

Sensor calibration for space translation

In various examples, a neural network may be trained for use in vehicle re-identification tasks—e.g., matching appearances and classifications of vehicles across frames—in a camera network. The neural network may be trained to learn an embedding space such that embeddings corresponding to vehicles of the same identify are projected closer to one another within the embedding space, as compared to vehicles representing different identities. To accurately and efficiently learn the embedding space, the neural network may be trained using a contrastive loss function or a triplet loss function. In addition, to further improve accuracy and efficiency, a sampling technique—referred to herein as batch sample—may be used to identify embeddings, during training, that are most meaningful for updating parameters of the neural network.

Training neural networks for vehicle re-identification

In various examples, sensor data—such as masked sensor data—may be used as input to a machine learning model to determine a confidence for object to person associations. The masked sensor data may focus the machine learning model on particular regions of the image that correspond to persons, objects, or some combination thereof. In some embodiments, coordinates corresponding to persons, objects, or combinations thereof, in addition to area ratios between various regions of the image corresponding to the persons, objects, or combinations thereof, may be used to further aid the machine learning model in focusing on important regions of the image for determining the object to person associations.

Determining associations between objects and persons using machine learning models

The present disclosure provides various approaches for smart area monitoring suitable for parking garages or other areas. These approaches may include ROI-based occupancy detection to determine whether particular parking spots are occupied by leveraging image data from image sensors, such as cameras. These approaches may also include multi-sensor object tracking using multiple sensors that are distributed across an area that leverage both image data and spatial information regarding the area, to provide precise object tracking across the sensors. Further approaches relate to various architectures and configurations for smart area monitoring systems, as well as visualization and processing techniques. For example, as opposed to presenting video of an area captured by cameras, 3D renderings may be generated and played from metadata extracted from sensors around the area.

Scenario recreation through object detection and 3D visualization in a multi-sensor environment

In various examples, training methods as described to generate a trained neural network that is robust to various environmental features. In an embodiment, training includes modifying images of a dataset and generating boundary boxes and/or other segmentation information for the modified images which is used to train a neural network.

Automatic labeling and segmentation using machine learning models

In various examples, sensor data&#x2014;such as masked sensor data&#x2014;may be used as input to a machine learning model to determine a confidence for object to person associations. The masked sensor data may focus the machine learning model on particular regions of the image that correspond to persons, objects, or some combination thereof. In some embodiments, coordinates corresponding to persons, objects, or combinations thereof, in addition to area ratios between various regions of the image corresponding to the persons, objects, or combinations thereof, may be used to further aid the machine learning model in focusing on important regions of the image for determining the object to person associations.

In various examples, cloud computing systems may store frames of video streams and metadata generated from the frames in separate data stores, with each type of data being indexed using shared timestamps. Thus, the frames of a video stream may be stored and/or processed and corresponding metadata of the frames may be stored and/or generated across any number of devices of the cloud computing system (e.g., edge and/or core devices) while being linked by the timestamps. A client device may provide a request or query to dynamically annotate the video stream using a particular subset of the metadata. In processing the request or query, the timestamps may be used to retrieve video data representing frames of the video stream and metadata extracted from those frames across the data stores. The retrieved metadata and video data may be used to annotate the frames for display on the client device.

Overlaying metadata on video streams on demand for intelligent video analysis

In various examples, a neural network may be trained for use in vehicle re-identification tasks&#x2014;e.g., matching appearances and classifications of vehicles across frames&#x2014;in a camera network. The neural network may be trained to learn an embedding space such that embeddings corresponding to vehicles of the same identify are projected closer to one another within the embedding space, as compared to vehicles representing different identities. To accurately and efficiently learn the embedding space, the neural network may be trained using a contrastive loss function or a triplet loss function. In addition, to further improve accuracy and efficiency, a sampling technique&#x2014;referred to herein as batch sample&#x2014;may be used to identify embeddings, during training, that are most meaningful for updating parameters of the neural network.

Smart area monitoring with artificial intelligence

Processing video for low-bandwidth transmission may be complex. At a content source embodiment of methods disclosed herein may include assigning the content identifier as a function of content in a packet of a packet stream on a packet-by-packet basis. The method may further comprise forwarding the content identifier with the packet to enable a downstream network node or device to effect prioritization of the packet within the packet stream. The downstream network node or device may make drop decisions that are guided by a content identifier. Packets, or video frames that contain useful information may be prioritized and have a higher probability of being delivered.

Method and apparatus for efficiently prioritizing elements in a video stream for low-bandwidth transmission

Systems and methods are described herein that provide a three-tier intelligent video surveillance management system. An example of a system described herein includes a gateway configured to obtain video content and metadata relating to the video content from a plurality of network devices, a metadata processing module communicatively coupled to the gateway and configured to filter the metadata according to one or more criteria to obtain a filtered set of metadata, a video processing module communicatively coupled to the gateway and the metadata processing module and configured to isolate video portions, of video the content, associated with respective first portions of the filtered set of metadata, and a cloud services interface communicatively coupled to the gateway, the metadata processing module and the video processing module and configured to provide at least some of the filtered set of metadata or the isolated video portions to a cloud computing service.

Cloud-based video surveillance management system

An automated security surveillance system determines a location of a possible disturbance and adjusts its cameras to record video footage of the disturbance. In one embodiment, a disturbance can be determined by recording audio of the nearby area. A system, coupled to a camera, may include an arrangement of four audio sensors recording audio of the nearby area to produce independent outputs. The system further may include a processing module configured to determine an angle and distance of an audio source relative to a location of the arrangement of the four audio sensors. The system can then adjust the camera by rotation along an azimuth or elevation angle and adjusting the zoom level to record video of the audio source. Through use of the system, a surveillance system can present an image of a source of possible disturbance to an operator more rapidly and precisely than through manual techniques.

System and method for audio source localization using multiple audio sensors

Growing community of inventors

Clovis, CA, United States of America

Farzin Aghdasi