Nec Corporation

Tonal Systems, Inc.

Nec C Orporation

Asim Kadav

Hans Peter Graf

Farley Lai

Erik Kruus

Renqiang Min

Igor Durdanovic

Aly E Orady

Gabriel Peal

Ivan Savytskyi

Rajiv Bhan

Hao Li

Iain Melvin

Giuseppe Barbalinardo

Joshua Ben Shapiro

Ning Xie

Rustam Paringer

Yi Huang

Srimat T Chakradhar

Biplob Kumar Debnath

Brandt Belson

Christopher Malon

Alexandru Niculescu-Mizil

Ryan Lafrance

Thomas Kroman Watt

Oliver Po

Michael Alexander Snower

Bryan James

Cheng-En Wu

Farhan Asif Chowdhury

Daniel Li

Likitha Lakshminarayanan

Chih-Yao Ma

Huahua Wang

Cun Mu

Juho Kim

Yizhe Zhu

Meera Hahn

Jie Chen

Huayu Li

Honglu Zhou

An exercise machine includes a cable. It further includes an interface to a moveable camera device coupled with the exercise machine. It further includes a processor configured to receive a cable-based measurement associated with an exercise performed by a user. The processor is further configured to receive, from the moveable camera device, video information associated with the exercise. The processor is further configured to provide a workout determination based at least in part on both the cable-based measurement and the video information received from the moveable camera device.

Camera device for workout

A method for model training and deployment includes training, by a processor, a model to learn video representations with a self-supervised contrastive loss by performing progressive training in phases with an incremental number of positive instances from one or more video sequences, resetting the learning rate schedule in each of the phases, and inheriting model weights from a checkpoint from a previous training phase. The method further includes updating the trained model with the self-supervised contrastive loss given multiple positive instances obtained from Cascade K-Nearest Neighbor mining of the one or more video sequences by extracting features in different modalities to compute similarities between the one or more video sequences and selecting a top-k similar instances with features in different modalities. The method also includes fine-tuning the trained model for a downstream task. The method additionally includes deploying the trained model for a target application inference for the downstream task.

Self-supervised multimodal representation learning with cascade positive example mining

A first video of a first individual performing an exercise movement is received, wherein the first video is associated with a first guidance label. A modified version of a video is generated at least in part by passing the first video to a pose data change model. The modified version of the video is associated with a second guidance label. A guidance classifier model is trained using the modified version of the video.

Synthesis of exercise guidance training data

A first output is received from a first hardware optical sensor. A second output is received from a second hardware sensor. Guidance is provided for a movement via a user interface, wherein the guidance is based at least in part on: the first output from the first hardware optical sensor; the second output from the second hardware sensor; and a model based at least in part on historical performance of the movement; and wherein at least one of the first output and the second output triggers a condition.

Exercise guidance using multi-modal data

Methods and systems of training a neural network include training a feature extractor and a classifier using a first set of training data that includes one or more base cases. The classifier is trained with few-shot adaptation using a second set of training data, smaller than the first set of training data, while keeping parameters of the feature extractor constant.

Few-shot action recognition

A method for augmenting video sequences in a video reasoning system is presented. The method includes randomly subsampling a sequence of video frames captured from one or more video cameras, randomly reversing the subsampled sequence of video frames to define a plurality of sub-sequences of randomly reversed video frames, training, in a training mode, a video reasoning model with temporally augmented input, including the plurality of sub-sequences of randomly reversed video frames, to make predictions over temporally augmented target classes, updating parameters of the video reasoning model by a machine leaning algorithm, and deploying, in an inference mode, the video reasoning model in the video reasoning system to make a final prediction related to a human action in the sequence of video frames.

Temporal augmentation for training video reasoning system

A computer-implemented method is provided for action localization. The method includes converting one or more video frames into person keypoints and object keypoints. The method further includes embedding position, timestamp, instance, and type information with the person keypoints and object keypoints to obtain keypoint embeddings. The method also includes predicting, by a hierarchical transformer encoder using the keypoint embeddings, human actions and bounding box information of when and where the human actions occur in the one or more video frames.

Keypoint based action localization

A surveillance system is provided. The surveillance system is configured for (i) detecting and tracking persons locally for each camera input video stream using the common area anchor boxes and assigning each detected ones of the persons a local track id, (ii) associating a same person in overlapping camera views to a global track id, and collecting associated track boxes as the same person moves in different camera views over time using a priority queue and the local track id and the global track id, (iii) performing track data collection to derive a spatial transformation through matched track box spatial features of a same person over time for scene coverage and (iv) learning a multi-camera tracker given visual features from matched track boxes of distinct people across cameras based on the derived spatial transformation.

Semi-automatic data collection and association for multi-camera tracking

An exercise machine accessory is disclosed. In one embodiment, a resistance identifier is configured to identify resistance for an exercise machine associated with the exercise machine accessory, wherein the resistance identifier is coupled to the exercise machine. In one embodiment, a motion identifier is configured to identify exercise motion for a user of the exercise machine. In one embodiment, a communications module is configured to communicate with the user of the exercise machine, wherein the communications module is coupled to the resistance identifier and the motion identifier.

Exercise machine resistance identifier

A method for using a multi-hop reasoning framework to perform multi-step compositional long-term reasoning is presented. The method includes extracting feature maps and frame-level representations from a video stream by using a convolutional neural network (CNN), performing object representation learning and detection, linking objects through time via tracking to generate object tracks and image feature tracks, feeding the object tracks and the image feature tracks to a multi-hop transformer that hops over frames in the video stream while concurrently attending to one or more of the objects in the video stream until the multi-hop transformer arrives at a correct answer, and employing video representation learning and recognition from the objects and image context to locate a target object within the video stream.

Multi-hop transformer for spatio-temporal reasoning and localization

Aspects of the present disclosure describe systems, methods and structures providing contextual grounding&#x2014;a higher-order interaction technique to capture corresponding context between text entities and visual objects.

Contextual grounding of natural language phrases in images

Aspects of the present disclosure describe systems, methods, and structures that provide action recognition with high-order interaction with spatio-temporal object tracking. Image and object features are organized into into tracks, which advantageously facilitates many possible learnable embeddings and intra/inter-track interaction(s). Operationally, our systems, method, and structures according to the present disclosure employ an efficient high-order interaction model to learn embeddings and intra/inter object track interaction across the space and time for AR. Each frame is detected by an object detector to locate visual objects. Those objects are linked through time to form object tracks. The object tracks are then organized and combined with the embeddings as the input to our model. The model is trained to generate representative embeddings and discriminative video features through high-order interaction which is formulated as an efficient matrix operation without iterative processing delay.

Action recognition with high-order interaction through spatial-temporal object tracking

A computer-implemented method executed by at least one processor for performing mini-batching in deep learning by improving cache utilization is presented. The method includes temporally localizing a candidate clip in a video stream based on a natural language query, encoding a state, via a state processing module, into a joint visual and linguistic representation, feeding the joint visual and linguistic representation into a policy learning module, wherein the policy learning module employs a deep learning network to selectively extract features for select frames for video-text analysis and includes a fully connected linear layer and a long short-term memory (LSTM), outputting a value function from the LSTM, generating an action policy based on the encoded state, wherein the action policy is a probabilistic distribution over a plurality of possible actions given the encoded state, and rewarding policy actions that return clips matching the natural language query.

Efficient and fine-grained video retrieval

Aspects of the present disclosure describe systems, methods and structures for an efficient multi-person posetracking method that advantageously achieves state-of-the-art performance on PoseTrack datasets by only using keypoint information in a tracking step without optical flow or convolution routines. As a consequence, our method has fewer parameters and FLOPs and achieves faster FPS. Our method benefits from our parameter-free tracking method that outperforms commonly used bounding box propagation in top-down methods. Finally, we disclose tokenization and embedding multi-person pose keypoint information in the transformer architecture that can be re-used for other pose tasks such as pose-based action recognition.

Keypoint based pose-tracking using entailment

Growing community of inventors

Jersey City, NJ, United States of America

Asim Kadav