Apple Inc.

Atila Orhon

Vignesh Jagadeesh

Marco Zuliani

This application relates to use of transformer neural networks to generate dynamic parameters for use in convolutional neural networks. In various embodiments, received image data is encoded and the encoded signal is sent to both a decoder and a transformer neural network. The decoder outputs a decoded data for input into a convolutional neural network. The transformer outputs a set of dynamic parameter values for input into the convolutional neural network. The convolutional neural network may use the decoded data and the set of dynamic parameter values to output instance image data show identifying a number of objects in an image. In various embodiments, the decoded data is also used to generate semantic data. The semantic data may be combined with the instance data to form panoptic image data.

Training transformer neural networks to generate parameters of convolutional neural networks

A first subset of pixels of an image may be labeled with an object identifier based on user interactions with the image. Pixel data representing the pixels of the image may be passed through an embedding neural network model to generate pixel embedding vectors. A prototype embedding vector associated with the object identifier may be generated based pixel embedding vectors corresponding to the first subset of pixels. For each pixel of a second subset of pixels of the image, a probability that the pixel should be labeled with the object identifier may be determined based on the prototype embedding vector and pixel embedding vectors corresponding to the second subset of pixels. Pixels of the second subset of pixels may be labeled with the object identifier based on the determined probabilities, and the pixels in the image may be segmented based on the pixels labeled with the object identifier.

Interactive image segmentation

Training a network for image processing with temporal consistency includes obtaining un-annotated frames from a video feed. A pretrained network is applied to the first frame of first frame set comprising a plurality of frames to obtain a first prediction, wherein the pretrained network is pretrained for a first image processing task. A current version of the pretrained network is applied to each frame of the first frame set to obtain a first prediction. A content loss term is determined, based on the first prediction and a current prediction for the frame, based on the current network. A temporal consistency loss term is also determined based on a determined consistency of pixels within each frame of the first frame set. The pretrained network may be refined based on the content loss term and the temporal term to obtain a refined network.

Growing community of inventors

Cupertino, CA, United States of America

Atila Orhon