Streamsets, Inc.

Alejandro Humberto Abdelnur

Arvind Prabhakar

Madhukar Devaraju

In various example embodiments, a system, computer readable medium and method for schema update engine dynamically updating a target data storage system. Incoming data records are received. A front-end schema of the incoming data records is identified. The front-end schema and the current target schema are compared. Based on identifying a difference between the front-end schema and the current target schema, the current target schema is updated in order to be identical to the front-end schema. The current target data file is closed and the incoming data records are stored in a new target data file according to the updated target schema.

Automatic drift detection and handling

Systems and methods herein describe accessing an original change data capture (CDC) dataset comprising information describing changes to a source database, the original CDC dataset comprising a plurality of entries; identifying a first entry of the plurality of entries comprising a primary-key, a first operation and entry data; identifying a set of entries in the plurality of entries that includes the primary-key; comparing the first operation of the first entry with a second operation of a second entry in the set of entries; updating the first operation and the entry data based on the comparison; generating a new entry based on the updating of the first operation and the entry data; storing the new entry in a consolidated CDC dataset; and applying the consolidated CDC dataset to a target database.

Growing community of inventors

Barcelona, Spain

Alejandro Humberto Abdelnur