Amazon Technologies, Inc.

Intelius, Inc.

Andrew Borthwick

Stephen Michael Ash

Timothy Jones

Shehzad Qureshi

Anurag Windlass Gupta

Mehul A Shah

Mehul Y Shah

Prajakta Datta Damle

Juliana Saussy

Abhishek Sharma

Pedrito Uriah Maynard-Zhang

Chethan Kommaranahalli Rudramuni

Adam Lawrence Joseph Heinermann

Alaykumar Navinchandra Desai

Sergei Dobroshinsky

Russell Reas

Yen Ling Adelene Sim

William P McNeill

Tianyi Lu

Sheng Chen

Robert Anthony Barton, Jr

Specified performance attributes may be used to configure machine learning transformations for ETL jobs. Performance attributes for a machine learning pipeline that applies a model to as part of a transformation for an ETL job may be used to configure a parameter in a stage of the machine learning pipeline. The configured stage may then be used when training the model. The trained machine learning pipeline may then be applied as part of a transformation operation included in an ETL job performed by the ETL system.

Using specified performance attributes to configure machine learning pipepline stages for an ETL job

Supervised partitioning is used to perform record matching. A request to identify matches between records is received. A graph representation that indicates similarities between the records is partitioned and an evaluation of the partitioning is performed according to a supervised machine learning technique to generate a confidence value in the partitioning. An indication of equivalent records according to the partitioning and the confidence value of the partitioning may be provided.

Supervised graph partitioning for record matching

Techniques for memory-efficient streaming count estimation for multisets are described. A method for memory-efficient streaming count estimation for multisets may include obtaining data from a plurality of data sources, and estimating a count for one or more attributes of the data using a telescoping count-min sketch (CMS) data structure, the telescoping CMS including at least a first table and a second table, wherein count values for the data are stored in a plurality of cells of the first table and when a cell of the first table is saturated, the count values for that cell are stored in a corresponding cell of the second table determined based at least on the cell of the first table.

Memory-efficient streaming count estimation for multisets

Using specified performance attributes to configure machine learning pipeline stages for an ETL job

Techniques for generating record embeddings from structured records are described. A record embeddings generating engine processes structured records to build a token vocabulary. Token embeddings are created for each token in the vocabulary. The token embeddings are trained using a loss function that relates the token embeddings to the record-attribute-data structure of the structured records. A record embedding is assembled from the trained token embeddings.

Unsupervised representation learning for structured records

Techniques for scaling record linkage via elimination of highly overlapped blocks are described. A method for scaling record linkage via elimination of highly overlapped blocks includes identifying a first plurality of blocks based at least on a plurality of records stored in a storage service of a provider network, identifying a plurality of sets of matching blocks from the first plurality of blocks, deleting the plurality of sets of matching blocks except for a first block from each set from the plurality of sets of matching blocks, and iteratively performing dynamic blocking based at least on the first block to generate subsequent pluralities of blocks until the subsequent pluralities of blocks are below a threshold size.

Scaling record linkage via elimination of highly overlapped blocks

Techniques for Scalable parallel elimination of approximately subsumed sets are described. A method for Scalable parallel elimination of approximately subsumed sets includes identifying a first plurality of blocks based at least on a plurality of records stored in a storage service of a provider network, determining a plurality of subsumption relationships between blocks from the first plurality of blocks, retaining a first subset of the first plurality of blocks and demoting a second subset of the first plurality of blocks based at least on the plurality of subsumption relationships, and iteratively performing dynamic blocking based at least on the first subset of the plurality of matching blocks and the second subset of the plurality of matching blocks to generate a subsequent pluralities of blocks.

Scalable parallel elimination of approximately subsumed sets

Block size reduction iterations are performed on a plurality of blocks of records until a block size criterion is met. An iteration comprises identifying, from a first collection of blocks, using one or more pivot operations, a set of combinations of oversized blocks such that at least one record belongs to all blocks of a combination. A new block comprising records that are members of each block of a first combination of the set is included in a second collection of blocks to be examined in a subsequent iteration. On at least one block created in an iteration, analysis operations are performed.

Intersection-based dynamic blocking

Record Linkage (RL) is the task of identifying two or more records referring to the same entity (e.g., a person, a company, etc.). RL models can be based on Cost Sensitive Alternating Decision Trees (ADTree), an algorithm that uniquely combines boosting and decision trees algorithms to create shorter and easier-to-interpret linking rules. These models can be naturally trained to operate at industrial precision/recall operating points, and the shorter output rules are so clear that it can effectively explain its decisions to non-technical users via score aggregation or visualization. The models significantly outperform other baselines on the desired industrial operating points, and the improved understanding of the model's decisions led to faster debugging and feature development cycles.

Cost-sensitive alternating decision trees for record linkage

Dynamic blocking determines which pairs of records in a data set should be examined as potential duplicates. Records are grouped together into blocks by shared properties that are indicators of duplication. Blocks that are too large to be efficiently processed are further subdivided by other properties chosen in a data-driven way. We demonstrate the viability of this algorithm for large data sets. We have scaled this system up to work on billions of records on an 80 node Hadoop cluster.

Growing community of inventors

Kirkland, WA, United States of America

Andrew Borthwick