Landing Ai

Dillon Laird

Kai Yang

Andrew Yan-Tak Ng

Mark William Sabini

Daniel Bibireata

Yu Qing Zhou

Shankaranand Jagadeesan

Yuxiang Zhang

Quinn Killough

Sebastian Rodriguez

Yan Liu

Francisco Matias Cuenca-Acuna

Pingyang He

Ankur Rawat

Leela Seshu Reddy Cheedepudi

Camilo Iral

Whitney Blodgett

Zeqi Qiu

Mingrui Zhang

Yazhou Cao

Junjie Guan

Aldrin Leal

Ramesh Sampath

Camilo Zapata

Juan Camilo Fernancez

Cristobal Silva

YuQing Zhou

Sanjay Bodhu

Whit Blodgett

Gregory Diamos

Sanjay Boddhu

A system or method for identifying mislabels in a training dataset. The labeled dataset includes multiple labeled samples, each of which is labeled with a ground-truth label. The labeled samples are divided into multiple training subset and hold-out test subset. For each training subset and corresponding hold-out test subset, a machine learning model is trained using the training subset, and the trained machine learning model is applied to the hold-out test subset to generate prediction labels for samples in the corresponding hold-out test subset. Each prediction label has a confidence score indicating a likelihood of the prediction label is correct. A pair of prediction label and ground truth label corresponding to a same sample is compared to determine whether there is a candidate mislabel. The system determines whether the candidate mislabel is a mislabel based in part on a confidence score of the prediction label.

Data centric mislabel detection

A model management system adaptively refines a training dataset for more effective visual inspection. The system trains a machine learning model using the initial training dataset and sends the trained model to a client for deployment. The deployment process generates outputs that are sent back to the system. The system determines that performance of predictions for noisy data points are inadequate and determines a cause of failure based on a mapping of the noisy data point to a distribution generated for the training dataset across multiple dimensions. The system determines a cause of failure based on an attribute of the noisy datapoint that deviates from the distribution of the training dataset and performs refinement towards the training dataset based on the identified cause of failure. The system retrains the machine learning model with the refined training dataset and sends the retrained machine learning model back to the client for re-deployment.

Model management system for improving training data through machine learning deployment

A model management system performs error analysis on results predicted by a machine learning model. The model management system identifies an incorrectly classified image outputted from a machine learning model and identifies using the Neural Template Matching (NTM) algorithm, an additional image correlated to the selected image. The system outputs correlated images based on a given image and a selection by a user through a user interface of a region of interest (ROI) of the given image. The region is defined by a bounding polygon input and the correlated images include features correlated to the features within the ROI. The system prompts a task associated with the additional image. The system receives a response that includes an indication that the additional image is incorrectly labeled and including a replacement label and instruct that the machine learning model be retrained using an updated training dataset that includes the replacement label.

Guided workflow for deep learning error analysis

Growing community of inventors

Santa Monica, CA, United States of America

Dillon Laird