Scale Ai, Inc.

Alexandr Wang

Nathaniel John Herman

Samuel Jacob Clearman

Akshat Bubna

Shariq Shahab Hashme

Liren Tu

Jeffrey Zhihong Li

James Lennon

One embodiment of the present invention sets forth a technique for evaluating labeled data. The technique includes selecting, from a set of labels for a data sample, a subset of the labels representing non-outliers in a distribution of values in the set of labels. The technique also includes aggregating the subset of the labels into a benchmark for the data sample. The technique further includes generating, based on a comparison between the benchmark and an additional label, a benchmark score associated with the data sample, and generating a set of performance metrics for labeling the data sample based on the benchmark score.

Automatic benchmarking of labeling tasks

Growing community of inventors

San Francisco, CA, United States of America

Alexandr Wang