Amazon Technologies, Inc.

Roshan Ram Rammohan

Srikanth Thirumalai

Madhu Madhava Kurup

Ismail Baha Tutar

Jeremy Calvert

Deept Kumar

Yohan Mammen

Sean Gwizdak

Jiahui Jiang

Sivaranjini Dharmalingam

Nathan Thomas Close

Shantanu Shailendrakumar Fauji

Technologies are disclosed herein for generating and utilizing machine-learning generated classifiers configured to identify document relationships. Manually-generated data is captured that indicates if documents in a document corpus have a relationship with one another, such as duplicates or variations. A determination may then be made as to whether a classifier is to be generated based on the duplicate decision data. If a classifier is to be generated, machine learning may be performed using training documents from the document corpus and the duplicate decision data to generate a classifier. The machine-learning generated classifier may then be utilized in a production environment to determine whether a new document is a duplicate of documents in the document corpus and/or to identify other relationships between documents in the document corpus, such as documents that are similar or are variations of one another.

Automated development and utilization of machine-learning generated classifiers

Technologies are disclosed for mapping documents to candidate duplicate documents in a document corpus. A bitset optimized inverted index is created for a document corpus. A document is received for which candidate duplicate documents in the document corpus are to be identified. The document is tokenized using adaptive tokenization. A determination made as to whether tokens in the document are represented in the bitset optimized inverted index. A list of candidate duplicate documents is created for tokens represented in the optimized inverted index utilizing in-memory bitsets that map tokens to documents that contain the tokens in the document corpus.

Optimized mapping of documents to candidate duplicate documents in a document corpus

Embodiments of a system and method for identifying structured data items lacking requisite information for rule-based duplicate detection are described. Embodiments may include generating a deficiency score for each of multiple structured data items including applying a set of rules based on duplicate detection techniques to each given structured data item in order to perform a comparison of the given structured data item to itself. The deficiency score of the given structured data item may be based on a result of the comparison. Embodiments may also include, based on the deficiency scores of the structured data items, identifying one or more deficient structured data items having less than a requisite quantity of information for performing duplicate detection on structured data items. Embodiments may also include identifying one or more key attributes missing from some of the one or more deficient structured data items and requesting those key attributes.

System and method for identifying structured data items lacking requisite information for rule-based duplicate detection

Growing community of inventors

Seattle, WA, United States of America

Roshan Ram Rammohan