Nex Team Inc.

Apple Inc.

On Loy Sung

Keng Fai Lee

Wing Hung Chan

Man Hon Chan

Qi Zhang

Wang Fai Ng

Long Mak

Tin Yin Lam

Jorge S Fino

Alan C Chu

Wang Chiu Chan

Oi Ying Angela Lee

Sing Yeung Lai

Daniel Dejos

Shing Fat Mak

Ka Ho Lau

Methods and systems for touchless control of a device include receiving at least one image; determining a first map, the first map including a function that generates a correspondence between data associated with the image to a feature space; determining a second map including a representation of a user in the image based on the first map; determining a third map comprising a field associated with the data in the image; determining a characteristic of the user based on one or more of the second map and the third map; and tracking the characteristic of the user to generate a historical data of the characteristic for performing the touchless control of the device.

Methods and systems for touchless control with a mobile device

A computer-implemented method for video processing is disclosed. The method includes receiving an input video of one or more persons from a camera; detecting a sequence of human poses in the input video using an artificial intelligence (AI) based technique; selecting a proper pose from among multiple poses in a given frame of the input video, to generate a sequence of proper poses; detecting one or more key points in the sequence of proper poses; computing changes in coordinates of the one or more key points; computing a function of the changes in the coordinates of the one or more key points in the sequence of proper poses; counting a given user movement as a repetitive motion of an activity based on the function; and computing a plurality of statistics about the activity based on the counting. In some embodiments, the activity is running, jogging, walking jumping, performing jumping jacks, squatting, and/or dribbling.

Video-based motion counting and analysis systems and methods for virtual fitness application

Systems and methods for motion tracking on a hardware device without a camera and/or without a sufficiently powerful hardware processor, by offloading pose processing fully or partially to a mobile device, are disclosed. Methods executable on the hardware device include receiving pose data associated with a user from the mobile device, and performing a touchless control of an application on the hardware device, based on the received pose data. Methods executable on the mobile device include capturing at least one image associated with a user, performing a pose recognition algorithm on the mobile device, based on the captured image, to generate pose data for performing a touchless control of an application on the hardware device, and transmitting the pose data to the hardware device, using the network connection. The methods and systems include embodiments where the pose recognition algorithm uses machine learning.

Methods and systems for offloading pose processing to a mobile device for motion tracking on a hardware device without a camera

Methods and systems for touchless control of a device include receiving an image from a camera and applying a machine learning technique the image to determine a first map, the first map including a function that generates a correspondence between data associated with the image to a feature space. A second map including a representation of a user in the image can be determined based on the first map. Further, a pose of the user can be determined based on the second map. The pose of the user and a portion of a body of the user can be tracked to determine a mapping between the portion of the user's body and an input feature of a device. The input feature can be modified based on the mapping to enable a touchless interaction between the user and the device.

A system and method for implementing a virtual fitness application are disclosed. Embodiments may be built for various platforms, including web browsers and mobile phones. One embodiment uses pose estimation to count the repetitive motions of a fitness activity (e.g., running). In one embodiment, to count the repetitive motions, a repetitive motion counting process is used that is based on computing differences of Y-coordinates of key points of the user. The repetitive motion counting process includes selecting a proper pose of the user; computing one or more delta values (corresponding to Y-coordinate changes of the key points); and counting a given user movement as a repetitive motion based on a function (e.g., average) of the delta values. One embodiment uses the pose estimation and repetitive motion count results to provide a gamified experience, for example, a leaderboard, a stats report, instant visual feedback, badges, coins, and a social experience.

Methods and systems for determining a ball shot attempt location on a ball court are disclosed. The method includes receiving an input video, where the input video includes the ball shot attempt and one or more players; receiving a declared ball shot attempt; determining a potential shooter by detecting a moving object flow in the input video; determining a potential shooter image region in the image plane by analyzing an overlapping image area of the ball trajectory and the potential shooter; determining a shooter foot-on-court time; determining a most probable pair of shooter legs in the potential shooter image region; determining a shooter foot location based on the most probable pair of shooter legs; and determining a ball shot attempt location on the ball court by applying a camera projection to the shooter foot location.

Methods and systems for determining ball shot attempt location on ball court

Methods and systems for constructing a three-dimensional (3D) model of a user in a virtual environment for full body virtual reality (VR) applications are described. The method includes receiving an image of the user captured using an RGB camera; detecting a body bounding box associated with the user using a first trained neural network; determining a segmentation map of the user, based on the body bounding box; determining a two-dimensional (2D) contour of the user from the segmentation map; forming a 3D extrusion model by extruding the 2D contour; and constructing the 3D model of the user in the virtual environment by applying a geometric transformation to the 3D extrusion model. Applications of full body VR include physical training and fitness sessions, games, control of computing devices, manipulation and display of data, interactive social media with VR, and the like.

Full body virtual reality utilizing computer vision from a single camera and associated systems and methods

Methods and systems for determining and classifying a number of repetitive motions in a video are described, and include the steps of first determining a plurality of images from a video, where the images are segmented from at least one video frame of the video. Next, performing a pose detection process on a feature of the images to generate one or more landmarks. Next, determining one or more principle component axes on points associated with a given landmark. Finally, determining at least one repetitive motion based on a pattern associated with a projection of the points onto the one or more principle components. In some embodiments, the disclosed methods can classify the repetitive motions to respective types. The present invention can be implemented for convenient use on a mobile computing device, such as a smartphone, for tracking exercises and similar repetitive motions.

Repetition counting and classification of movements systems and methods

Methods and systems for determining a ball shot attempt result, the ball shot attempt captured by a camera on a mobile computing device, are disclosed. The methods and systems perform steps comprising receiving an input video captured using the camera on the mobile computing device, wherein the input video comprises a ball shot attempt at a goal; receiving a declared ball shot attempt, the declared ball shot attempt detected by analyzing a ball trajectory in the input video; detecting an element movement associated with an element of the goal; and determining the ball shot attempt result based on the element movement and the ball trajectory. In some embodiments, the element of the goal is a net, and the declared ball shot attempt is detected by analyzing the ball trajectory in a ball shot attempt Region of Interest (ROI) within frames of the input video.

Methods and systems for ball game analytics with a mobile device

Methods and systems for detecting a player in a gaming environment using a mobile device are disclosed. The methods and systems perform the steps of receiving calibration data from a camera of a user device, the calibration data comprising a first reference point associated with the gaming environment, a second reference point associated with the gaming environment, and sensor data associated with the user device; computing a camera projection based on the first reference point, the second reference point, and the sensor data obtained from the user device; receiving an input video of the gaming environment, wherein the input video is captured using the camera, and wherein frames of the input video comprise at least one player in the gaming environment; identifying an image location of the player by performing a computer vision algorithm on the input video; and determining a real-world location of the player based on the image location and the camera projection.

Methods and systems for player location determination in gameplay with a mobile device

Methods and systems for mobile device-based touchless control can include receiving an image of a user and applying a convolutional neural network to the image to determine a feature map. Further, a key-points map and an affinity-field map can be determined based on the feature map, and the pose of the user can be determined based on the maps. The methods and system further include identifying an activation gesture of the user and determining a validity of a bounding box associated with the pose of the user, and tracking the pose of the user and a portion of a body of the user for performing the touchless control. Further, the methods and systems include determining a mapping between the portion of the user's body and a cursor position associated with an application, and moving the cursor position to enable touchless user interaction. Embodiments use computer vision techniques to enable a resource-limited mobile device to conduct the aforementioned steps.

Methods and systems for ball shot attempt detection and game analytics generation are disclosed. The methods and systems perform steps to receive an input video of a ball gameplay, wherein the input video was captured using a stationary camera, and wherein frames of the input video comprises a goal; identify a Region of Interest (ROI) surrounding the goal by performing a first computer vision algorithm on the input video; detect a ball within the ROI during a shot attempt and determining a trajectory of the ball by performing a second computer vision algorithm on the input video; and identify a player relevant to the shot attempt based on the trajectory of the ball. In some embodiments, the computer vision algorithms comprise a convolution neural network (CNN). The present invention uses computer vision techniques to enable a resource-limited mobile device such as a smartphone to efficiently execute the new process.

Various features and processes related to document collaboration are disclosed. In some implementations, animations are presented when updating a local document display to reflect changes made to the document at a remote device. In some implementations, a user can selectively highlight changes made by collaborators in a document. In some implementations, a user can select an identifier associated with another user to display a portion of a document that includes the other user's cursor location. In some implementations, text in document chat sessions can be automatically converted into hyperlinks which, when selected, cause a document editor to perform an operation.

Smart text in document chat

Method and apparatus for color management of documents retrieved and manipulated by a server based application and displayed on a client device through a browser application. Page elements of documents are manipulated to provide consistent and/or accurate color reproduction on the client device. Pages include untagged images, images tagged with color profiles, and tagged or untagged page elements specified by hex color values. Color values for images and untagged page elements specified by hex color values are adjusted by the server application before transmission to the client device to account for properties of browser applications and operating systems of different client devices.

Growing community of inventors

Hong Kong, China

On Loy Sung