Optimization of feature embeddings for deep learning models

Abstract

A system trains models to generate embeddings that represent likelihoods associated with features. For example, an embedding may be generated for users and pages such that a user's embedding represents how likely a user is to comment on a given page. Initially, memory space for storing each embedding may be overprovisioned. The system monitors the embeddings for a feature as they are generated and recalculated over time. If the system detects that a particular index value is never updated for embeddings of that feature, then the system may remove that value from the feature embeddings. This allows the array lengths of embeddings to be customized to the particular features they represent, saving memory space. The system may further use related information to identify pooling functions that are most effective for particular features, to identify similarities between entities, and to provide insight into how the feature data influences neural network layers.