Amazon Technologies, Inc.

Microsoft Technology Licensing, LLC

Igor Kviatkovsky

Gerard Guy Medioni

Manoj Aggarwal

Alon Shoshan

Eyal Krupka

Nadav Israel Bhonker

Igor Abramovski

Dilip Kumar

Lior Zamir

Ehud Benyamin Rivlin

Amit Adam

Miriam Farber

Jinyu Li

Jason Matthew Cahill

Cha Zhang

Tommer Leyvand

Roman Goldenberg

Jordan Tyler Williams

Timothy R O'Connor

Tim Keosababian

Yevgeni Nogin

Gal Novich

Shunit Haviv

Ori Linial

Oron Anschel

Shunit Haviv Hakimi

Adam Botach

Divyansh Aggarwal

During operation a biometric identification system acquires an image of at least a portion of a user that is processed by a representation model to determine representation data. The representation data may be securely stored for later use. The image is not stored. The stored representation data is smaller than equivalent image data. The representation model may comprise a subset of a first recognition model that has been trained using image training data. The image training data may be processed by the trained representation model to determine training representation data. The representation data contains sufficient information to be used as training data for recognition models, instead of acquired images. Training of recognition models using representation data is more computationally efficient compared to training with image data. Recognition comparisons using query representation data are more computationally efficient compared to query image data.

Storable representation of biometric data

An image of at least a portion of a user during enrollment to a biometric identification system is acquired and processed with a plurality of models to determine embeddings that are representative of features in that image in respective embedding spaces. Transformers are trained to accept as input an embedding from a respective model and produce as output a transformed embedding consistent with a common embedding space. These transformed embeddings are aggregated to produce transformed aggregated embedding data and stored for later comparison to identify the user. Subsequently, one or more embedding models may be used to process a query image to determine a query embedding. The query embedding may be transformed into the common embedding space for comparison.

System to reduce data retention using aggregated embedding data

A biometric identification system acquires an image of at least a portion of a user that is processed by an embedding model. Additional training of the embedding model is performed using selected training data. In one implementation, the selected training data is determined based on similarity to embedding data acquired during operation. For example, a query image is processed to determine query embedding data. If the query embedding is deemed to be a close match to embedding data of a previously enrolled user, additional synthetic training data that is also similar to the query embedding data is selected for use in subsequent training. In another implementation, enrolled user data may be analyzed to find close matches to synthetic training data, from which selected training data is determined. Use of the additional synthetic training data facilitates the subsequent training in distinguishing between similar instances of embedding data.

System to determine biometric identification training data

A biometric identification system may store biometric data for later assessment. Data storage parameters, such as cryptographic keys used to encrypt and decrypt the biometric data, may be determined based on the biometric data. In one implementation, the biometric data comprises embedding data in an embedding space. During enrollment and storage, the embedding data is assessed to determine nearest anchor data in the embedding space. Cryptographic parameters, such as an encryption key, are determined based on 'k' anchor data that are within a threshold distance of the embedding data in the embedding space. During query, query embedding data is similarly processed to determine cryptographic parameters, such as a decryption key. The decryption key may then be used to attempt decryption of the encrypted at-rest biometric data. If successful, the decrypted biometric data may then be compared to the query embedding to assert an identity.

System for determining storage parameters for biometric data

A user performs an enrollment process to utilize a biometric identification system. This includes acquisition of biometric input data. Accuracy of subsequent identification is improved by utilizing high quality input data during enrollment. Input data is processed using a plurality of embedding models to determine a plurality of embedding vectors. These embedding vectors are translated into a common embedding space. Input quality may be determined based on analysis of these embedding vectors. For example, if a mean distance of the translated embedding vectors is less than a threshold value, the input data may be deemed to be of sufficient quality for use to complete an enrollment process. This analysis may also be used for post-enrollment operation, such as during an identification process to determine query input data that is of insufficient quality.

System to assess biometric input data

Images of a hand are obtained by a camera. A pose of the hand relative to the camera may vary due to rotation, translation, articulation of joints in the hand, and so forth. Avatars comprising texture maps from images of actual hands and three-dimensional models that describe the shape of those hands are manipulated into different poses and articulations to produce synthetic images. Given that the mapping of points on an avatar to the synthetic image is known, highly accurate annotation data is produced that relates particular points on the avatar to the synthetic image. An artificial neural network (ANN) is trained using the synthetic images and corresponding annotation data. The trained ANN processes a first image of a hand to produce a second image of the hand that appears to be in a standardized or canonical pose. The second image may then be processed to identify the user.

System for mapping images to a canonical space

An image of at least a portion of a user during enrollment to a biometric identification system is acquired and processed with a first model to determine a first embedding that is representative of features in that image in a first embedding space. The first embedding may be stored for later comparison to identify the user, while the image is not stored. A second model that uses a second embedding space may be later developed. A transformer is trained to accept as input an embedding from the first model and produce as output an embedding consistent with the second embedding space. The previously stored first embedding may be converted to a second embedding in a second embedding space using the transformer. As a result, new embedding models may be implemented without requiring storage of user images for later reprocessing with the new models or requiring re-enrollment by users.

System to reduce data retention

Images of a hand are obtained by a camera. These images may depict the fingers and palm of the user. A pose of the hand relative to the camera may vary due to rotation, translation, articulation of joints in the hand, and so forth. One or more canonical images are generated by mapping the images to a canonical model. A first embedding model is used to determine a first embedding vector representative of the palm as depicted in the canonical images. A second embedding model is used to determine a set of second embedding vectors, each representative of individual fingers as depicted in the canonical images. Embedding distances in the embedding space from the embedding vectors to a closest match of previously stored embedding vectors are multiplied together to determine an overall distance. If the overall distance is less than a threshold value, an identity of a user is asserted.

System for biometric identification

A machine learning system to determine an identity of a user is trained using triplets of ad hoc synthetic data and actual data. The data may comprise multimodal images of a hand. Each triplet comprises an anchor, a positive, and a negative image. Synthetic triplets for different synthesized identities are generated on an ad hoc basis and provided as input during training of the machine learning system. The machine learning system uses a pairwise label-based loss function, such as a triplet loss function during training. Synthetic triplets may be generated to provide more challenging training data, to provide training data for categories that are underrepresented in the actual data, and so forth. The system uses substantially less memory during training, and the synthetic triplets need not be retained further reducing memory use. Ongoing training is supported as new actual triplets become available, and may be supplemented by additional synthetic triplets.

System for training of recognition system using ad hoc training data

A scanner acquires a set of images of a hand of a user to facilitate identification. These images may vary, due to changes in relative position, pose, lighting, obscuring objects such as a sleeve, and so forth. A first neural network determines output data comprising a spatial mask and a feature map for individual images in the set. The output data for two or more images is combined to provide aggregate data that is representative of the two or more images. The aggregate data may then be processed using a second neural network, such as convolutional neural network, to determine an embedding vector. The embedding vector may be stored and associated with a user account. At a later time, images acquired from the scanner may be processed to produce an embedding vector that is compared to the stored embedding vector to identify a user at the scanner.

System for determining embedding from multiple inputs

During a training phase, a first machine learning system is trained using actual data, such as multimodal images of a hand, to generate synthetic image data. During training, the first system determines latent vector spaces associated with identity, appearance, and so forth. During a generation phase, latent vectors from the latent vector spaces are generated and used as input to the first machine learning system to generate candidate synthetic image data. The candidate image data is assessed to determine suitability for inclusion into a set of synthetic image data that may be used for subsequent use in training a second machine learning system to recognize an identity of a hand presented by a user. For example, the candidate synthetic image data is compared to previously generated synthetic image data to avoid duplicative synthetic identities. The second machine learning system is then trained using the approved candidate synthetic image data.

System for synthesizing data

Described is a method for processing image data to determine if a portion of the imaged environment is exposed to high illumination, such as sunlight. In some implementations, image data from multiple different imaging devices may be processed to produce for each imaging device a respective illumination mask that identifies pixels that represent a portion of the environment that is exposed to high illumination. Overlapping portions of those illumination masks may then be combined to produce a unified illumination map of an area of the environment. The unified illumination map identifies, for different portions of the environment, a probability that the portion is actually exposed to high illumination.

Image data and simulated illumination maps

A user's collection of images may be analyzed to identify people's faces within the images, then create clusters of similar faces, where each of the clusters may represent a person. The clusters may be ranked in order of size to determine a relative importance of the associated person to the user. The ranking may be used in many social networking applications to filter and present content that may be of interest to the user. In one use scenario, the clusters may be used to identify images from a second user's image collection, where the identified images may be pertinent or interesting to the first user. The ranking may also be a function of user interactions with the images, as well as other input not related to the images. The ranking may be incrementally updated when new images are added to the user's collection.

Growing community of inventors

Haifa, Israel

Igor Kviatkovsky