Hitachi Vantara Corporation

Io-tahoe Llc.

Rohit Mahajan

Alex Mylnikov

Winnie W Cheng

Tomoya Wada

Ram Dayal Goyal

Winnie Cheng

A system and method for continuous processing of data streams residing in distributed data sources. The method includes: receiving a plurality of data streams from a plurality of distributed data sources; processing each of the plurality of data streams using a plurality commands, the plurality commands are executed in parallel using a graph execution engine; and transporting, through a transport layer, each of the plurality of data streams using the plurality commands to at least one data sink.

Computer implemented method for continuous processing of data-in-motion streams residing in distributed data sources

A system and method for data entries deduplication are provided. The method includes indexing an input data set, wherein the input data set is in a tabular formant and the indexing includes providing a unique Row identifier (RowID), wherein rows are the data entries; computing attribute similarity for each column across each pair of rows; computing, for each pair of rows, row-to-row similarity as a weighted sum of attribute similarities; clustering pairs of rows based on their row-to-row similarities; and providing an output data set including at least the clustered pairs of rows.

Techniques for database entries de-duplication

A system and method for performing a hash bucketing process on data in motion are presented. The method includes applying a first hash function on an input dataset to map the input dataset to a bucket, wherein the first hash function results with a first hash value; applying a second hash function on the first hash value to map the input dataset to a record in the bucket; generating metadata based on the input dataset, wherein the metadata at least points to the original location of the input dataset; and storing the generated metadata in the record in the bucket.

Techniques for processing of data in motion

A system and computer implemented method for cataloging database metadata using a signature matching process are provided. The method includes receiving an input name to be matched to a key in a seed table; generating a first fingerprint by decomposing the received input name into a first set n-grams; generating, based on the received input name, a second fingerprint using a predetermined pronunciation schema, wherein the second fingerprint is a phonetic fingerprint; generating a third fingerprint by decomposing the second fingerprint into a second set of n-grams; identifying a matching key by matching any combination of the first fingerprint, the second fingerprint, and the third fingerprint against keys in the seed table; and cataloging the metadata with the matching key as a tag.

Cataloging database metadata using a signature matching process

A method and system for tagging database columns are presented. The method includes receiving an input column name of at least one column in a database; performing signature matching of the input column name to contents of a seed table; determining a first confidence score for the signature matching; and tagging a matching value in the seed table as a tag for the input column name, when a first confidence score exceeds a first threshold value.

System and method for tagging database properties

A system and computer implemented method for cataloging database metadata using a probabilistic signature matching process are provided. The method includes receiving an input name to be matched to keys in a data corpus; dividing the received input name into a plurality of text segments; identifying a set of matching keys by matching each of the plurality text segments against keys in the data corpus; analyzing the set of matching keys to construct a tag; and cataloging the metadata with the matching key as the construct tag.

Cataloging database metadata using a probabilistic signature matching process

A method and an inclusion dependency determination system (IDDS) for determining inclusion dependency between columns of tables in a target database to establish primary key (PK)-foreign key (FK) relationships among data in the columns with minimized disk input and output operations are provided. The IDDS determines dependency characteristic data (DCD) of each column and arranges the columns by applying one or more predefined rules to the columns based on a minimum value of the data of each column. The IDDS determines pairs of arranged columns that demonstrate a possibility of inclusion dependency based on the DCD and identifies a first column and a second column of each determined pair as a candidate PK and a candidate FK respectively. The IDDS determines inclusion dependency between the candidate PK and the candidate FK on comparing data of the candidate PK with the data of the candidate FK using dynamically determined search techniques.

Growing community of inventors

Iselin, NJ, United States of America

Rohit Mahajan