Microsoft Technology Licensing, LLC

Liwen Zhang

Yi Zheng

Yongzheng Zhang

Weidong Zhang

Chi-Yi Kuan

Yurong Shi

Weiqin Ma

Bin Wu

SangHyun Park

Hong Lu

Srihari R Duddukuru

The disclosed embodiments provide a system for processing data. During operation, the system obtains a first configuration for processing a first set of content items from a first data source and a second configuration for processing a second set of content items from a second data source. For each content item in the first set of content items, the system uses mappings from the first configuration to transform original fields from the content item into required fields in a record representing the content item. Next, the system generates, from the required fields, a document key for the content item. The system also performs deduplication of multiple records with the document key and stores a single record with the document key. Finally, the system uses the second configuration to generate, from the second set of content items, a set of records independently of processing the first set of content items.

Scalable processing of heterogeneous user-generated content

Growing community of inventors

San Carlos, CA, United States of America

Liwen Zhang