Netskope, Inc.

Jason B Bryslawskyj

Yi Zhang

Yihua Liao

Ari Azarafrooz

Niranjan Koduri

Emanoel Daryoush

Wayne Xin

Dong Guo

Rongrong Tao

A network security system that classifies webpages and uses the classification of the webpages to enforce relevant security policies is disclosed. To classify a webpage, a screenshot of at least a portion of the webpage is captured. An embedding engine generates a subject image embedding of the screenshot. The subject image embedding is classified by an image classifier that includes an index of training image embeddings each having a label of their classification and an associated approximate nearest neighbors model trained to identify the label of the closest training image embedding to the subject image embedding and a score representing their similarity. The webpage is classified based at least in part on the score and the label from the image classifier. The network security system applies security policies to requests from client devices that identify the webpage as its destination based at least in part on the webpage classification.

Webpage categorization based on image classification of webpage screen capture

Image fingerprints (embeddings) are generated by an image fingerprinting model and indexed with an approximate nearest neighbors (ANN) model trained to identify the most similar fingerprint based on a subject embedding. For image matching, a score is provided that indicates a similarity between the input embedding and the most similar identified embedding, which allows for matching even when an image has been distorted, rotated, cropped, or otherwise modified. For image classification, the embeddings in the index are clustered and the clusters are labeled. Users can provide just a few images to add to the index as a labeled cluster. The ANN model returns a score and label of the most similar identified embedding for labeling the subject image if the score exceeds a threshold. As improvements are made to the image fingerprinting model, a converter model is trained to convert the original embeddings to be compatible with the new embeddings.

Image classification and accelerated classification training using deep learning image fingerprinting models and indexed embeddings

A system to generate an image classifier and test it nearly instantaneously is described herein. Image embeddings generated by an image fingerprinting model are indexed and an associated approximate nearest neighbors (ANN) model is generated. The embeddings in the index are clustered and the clusters are labeled. Users can provide just a few images to add to the index as a labeled cluster. The ANN model is trained to receive an image embedding as input and return a score and label of the most similar identified embedding. The label may be applied if the score exceeds a threshold value. The image classifier can be tested efficiently using Leave One Out Cross Validation ('LOOCV') to provide near-instantaneous quality indications of the image classifier to the user. Near-instantaneous indications of outliers in the provided images can also be provided to the user using a distance to the centroid calculation.

Image classifier automated testing and outlier detection

Image fingerprinting conversion between different image fingerprinting models

Image matching using deep learning image fingerprinting models and embeddings

Growing community of inventors

San Diego, CA, United States of America

Jason B Bryslawskyj