Apple Inc.

Xiaojin Shi

Hsi-Jung Wu

James O Normile

Francesco Rossi

Marco Zuliani

Xiaosong Zhou

Haitao Guo

Feng Tang

Alexey Marinichev

Barin Geoffry Haskell

Thomas Pun

Thorsten Gernoth

Xin Tong

Bartlomiej W Rymkowski

Yuxin Liu

Chong Chen

Chris Yoochang Chung

Gaurav Kapoor

Umesh Suresh Vaishampayan

Kit-Man Wan

Jim Normile

Gianluca Filippini

Cecile Marie Foret

Xiaohuan Corina Wang

Ionut Hristodorescu

Benjamin Poulain

Etienne Belanger

Albert Antony

Guy Cote

Athanasios Leontaris

Joe S Abuan

Hyeonkuk Jeong

Timothy S Paek

Vignesh Jagadeesh

Vinay Sharma

Minwoo Jeong

Keith P Avery

Brian Edward Walsh

Brandt M Westing

Jamil Dhanani

Omid Khalili

Ang Li

Harveen Kaur

Guillaume Tartavel

Brian P Keene

Barin Geoffrey Haskell

The subject technology provides a framework for evaluating activation functions of a neural network using lookup tables. In order to provide lookup table based activation functions with a desired precision within hardware constraints for the lookup tables, multiple lookup tables for each activation function can be provided. Each of the multiple lookup tables may correspond to a respective subrange of input values, within a full range of input values for the activation function.

Lookup table activation functions for neural networks

The subject technology provides receiving a neural network (NN) model to be executed on a target platform, the NN model including multiple layers that include operations and some of the operations being executable on multiple processors of the target platform. The subject technology further sorts the operations from the multiple layers in a particular order based at least in part on grouping the operations that are executable by a particular processor of the multiple processors. The subject technology determines, based at least in part on a cost of transferring the operations between the multiple processors, an assignment of one of the multiple processors for each of the sorted operations of each of the layers in a manner that minimizes a total cost of executing the operations. Further, for each layer of the NN model, the subject technology includes an annotation to indicate the processor assigned for each of the operations.

Compiling models for dedicated hardware

A method includes receiving input data at a trained machine learning model that includes a common part and task-specific parts, receiving an execution instruction that identifies one or more processing tasks to be performed, processing the input data using the common part of the trained machine learning model to generate intermediate data, and processing the intermediate data using one or more of the task-specific parts of the trained machine learning model based on the execution instruction to generate one or more outputs.

Machine learning model with conditional execution of multiple processing tasks

Intelligent systems are disclosed that respond to user intent and desires based upon activity that may or may not be expressly directed at the intelligent system. In some embodiments, the intelligent system acquires a depth image of a scene surrounding the system. A scene geometry may be extracted from the depth image and elements of the scene may be monitored. In certain embodiments, user activity in the scene is monitored and analyzed to infer user desires or intent with respect to the system. The interpretation of the user's intent as well as the system's response may be affected by the scene geometry surrounding the user and/or the system. In some embodiments, techniques and systems are disclosed for interpreting express user communication, e.g., expressed through hand gesture movements. In some embodiments, such gesture movements may be interpreted based on real-time depth information obtained from, e.g., optical or non-optical type depth sensors.

Multi media computing or entertainment system for responding to user presence and activity

Artistic styles extracted from source images may be applied to target images to generate stylized images and/or video sequences. The extracted artistic styles may be stored as a plurality of layers in one or more neural networks, which neural networks may be further optimized, e.g., via the fusion of various elements of the networks' architectures. The artistic style may be applied to the target images and/or video sequences using various optimization methods, such as the use of a first version of the neural network by a first processing device at a first resolution to generate one or more sets of parameters (e.g., scaling and/or biasing parameters), which parameters may then be mapped for use by a second version of the neural network by a second processing device at a second resolution. Analogous multi-processing device and/or multi-network solutions may also be applied to other complex image processing tasks for increased efficiency.

Enhanced image processing techniques for deep neural networks

The subject technology receives a neural network model in a model format, the model format including information for a set of layers of the neural network model, each layer of the set of layers including a set of respective operations. The subject technology generates neural network (NN) code from the neural network model, the NN code being in a programming language distinct from the model format, and the NN code comprising a respective memory allocation for each respective layer of the set of layers of the neural network model, where the generating comprises determining the respective memory allocation for each respective layer based at least in part on a resource constraint of a target device. The subject technology compiles the NN code into a binary format. The subject technology generates a package for deploying the compiled NN code on the target device.

Compiling code for a machine learning model for execution on a specialized processor

Artistic styles extracted from one or more source images may be applied to one or more target images, e.g., in the form of stylized images and/or stylized video sequences. The extracted artistic style may be stored as a plurality of layers in a neural network, which neural network may be further optimized, e.g., via the fusion of various elements of the network's architectures. An optimized network architecture may be determined for each processing environment in which the network will be applied. The artistic style may be applied to the obtained images and/or video sequence of images using various optimization methods, such as the use of scalars to control the resolution of the unstylized and stylized images, temporal consistency constraints, as well as the use of dynamically adjustable or selectable versions of Deep Neural Networks (DNN) that are responsive to system performance parameters, such as available processing resources and thermal capacity.

Flexible resolution support for image and video style transfer

Real-time selection of DNN style transfer networks from DNN sets

A video decoder system includes a video decoding engine, noise database, artifact estimator and post-processing unit. The video coder may generate recovered video from a data stream of coded video data, which may have visually-perceptible artifacts introduced as a byproduct of compression. The noise database may store a plurality of previously developed noise patches. The artifact estimator may estimate the location of coding artifacts present in the recovered video and select noise patches from the database to mask the artifacts and the post-processing unit may integrate the selected noise patches into the recovered video. In this manner, the video decoder may generate post-processed noise which may mask artifacts that otherwise would be generated by a video coding process.

Video processing for masking coding artifacts using dynamic noise maps

Varying embodiments of intelligent systems are disclosed that respond to user intent and desires based upon activity that may or may not be expressly directed at the intelligent system. In some embodiments, the intelligent system acquires a depth image of a scene surrounding the system. A scene geometry may be extracted from the depth image and elements of the scene, such as walls, furniture, and humans may be evaluated and monitored. In certain embodiments, user activity in the scene is monitored and analyzed to infer user desires or intent with respect to the system. The interpretation of the user's intent or desire as well as the system's response may be affected by the scene geometry surrounding the user and/or the system. In some embodiments, techniques and systems are disclosed for interpreting express user communication, for example, expressed through fine hand gesture movements. In some embodiments, such gesture movements may be interpreted based on real-time depth information obtained from, for example, optical or non-optical type depth sensors. The depth information may be interpreted in 'slices' (three-dimensional regions of space having a relatively small depth) until one or more candidate hand structures are detected. Once detected, each candidate hand structure may be confirmed or rejected based on its own unique physical properties (e.g., shape, size and continuity to an arm structure). Each confirmed hand structure may be submitted to a depth-aware filtering process before its own unique three-dimensional features are quantified into a high-dimensional feature vector. A two-step classification scheme may be applied to the feature vectors to identify a candidate gesture (step 1), and to reject candidate gestures that do not meet a gesture-specific identification operation (step-2). The identified gesture may be used to initiate some action controlled by a computer system.

In the field of Human-computer interaction (HCI), i.e., the study of the interfaces between people (i.e., users) and computers, understanding the intentions and desires of how the user wishes to interact with the computer is a very important problem. The ability to understand human gestures, and, in particular, hand gestures, as they relate to HCI, is a very important aspect in understanding the intentions and desires of the user in a wide variety of applications. In this disclosure, a novel system and method for three-dimensional hand tracking using depth sequences is described. Some of the major contributions of the hand tracking system described herein include: 1.) a robust hand detector that is invariant to scene background changes; 2.) a bi-directional tracking algorithm that prevents detected hands from always drifting closer to the front of the scene (i.e., forward along the z-axis of the scene); and 3.) various hand verification heuristics.

Three-dimensional hand tracking using depth sequences

A block input component of a video encoding pipeline may, for a block of pixels in a video frame, compute gradients in multiple directions, and may accumulate counts of the computed gradients in one or more histograms. The block input component may analyze the histogram(s) to compute block-level statistics and determine whether a dominant gradient direction exists in the block, indicating the likelihood that it represents an image containing text. If text is likely, various encoding parameter values may be selected to improve the quality of encoding for the block (e.g., by lowering a quantization parameter value). The computed statistics or selected encoding parameter values may be passed to other stages of the pipeline, and used to bias or control selection of a prediction mode, an encoding mode, or a motion vector. Frame-level or slice-level parameter values may be generated from gradient histograms of multiple blocks.

Encoding blocks in video frames containing text using histograms of gradients

An error recovery method may be engaged by an encoder to recover from misalignment between reference picture caches at the encoder and decoder. When a communication error is detected between a coder and a decoder, a number of non-acknowledged reference frames present in the decoder's reference picture cache may be estimated. Thereafter, frames may be coded as reference frames in a number greater or equal to the number of non-acknowledged reference frames that are estimated to be present in the decoder's reference picture cache. Thereafter, ordinary coding operations may resume. Typically, a final reference frame that is coded in the error recovery mode will be coded as a synchronization frame that has high coding quality. The coded reference frames that precede it may be coded at low quality (or may be coded as SKIP-coded frames). On reception and decoding, the preceding frames may cause the decoder to flush from its reference picture cache any non-acknowledged reference frames that otherwise might collide with the new synchronization frame. In this manner, alignment between the encoder and decoder may be restored.

Sync frame recovery in real time video transmission system

Growing community of inventors

Fremont, CA, United States of America

Xiaojin Shi