At&t Intellectual Property I, L.p.

Vladislav Shkapenyuk

Theodore Donald Johnson

Oliver Spatscheck

Divesh Srivastava

Marios Hadjieleftheriou

Yaron Kanza

Alexandre Gerber

Shanmugavelayutham Muthukrishnan

Frederick True

Lukasz Golab

Tamraparni Dasu

Deborah F Swayne

Graham Cormode

Pramod Jamkhedkar

Laks V S Lakshmanan

Bojian Xu

Architectures and techniques are presented that can more efficiently update derived data products in response to updated source data. Source data is typically stored in source tables, whereas a materialized view of a query can generate a derived table based on the state of the source tables at the time the query is executed. When source data changes (e.g., in response to late-arriving input data), rather than recomputing the entire derived table (e.g., by again executing the original query, which can be expensive), an invertible relationship between timestamps can be leveraged to identify only those portions of the derived table that are affected by the update. Therefore, a new defining query can be generated to update only those portions of the derived table that are affected by the source data update.

Update propagation in a data stream warehouse

Systems and methods utilize a data model database which includes a plurality of symbol data types. Each of the plurality of symbol data types have one or more symbol data fields. The data model database further includes a plurality of concrete data types. Each of the concrete data types have one or more language-agnostic concrete fields associated with each of the one or more symbol data fields. Each of the one or more language-agnostic concrete fields apply one or more concrete constraints to each of the corresponding symbol data fields. The data model database further includes a plurality of carrier data types. The plurality of carrier data types having one or more language-specific carrier fields associated with each of the one or more language-agnostic concrete fields. Each of the one or more language-specific carrier fields apply one or more carrier constraints to each of the corresponding language-agnostic concrete fields.

Data model database

A system may include a processor, a user input, and memory comprising a graph and executable instructions. The executable instructions may cause the processor to effectuate operations. The operations include receiving, via the user input, a query comprising a class generalization and pathway variables. The operations include identifying a query class based on at least the class generalization and determining an anchor set based on at least one of the pathway variables. The operations also include translating the pathway variables into a pathway algebraic expression based on the anchor set and the query class and executing the pathway algebraic expression on the graph to return a pathway set.

Concept-based querying of graph databases

A device, method and computer-readable medium for generating unique identification for records in a data streaming processing system are disclosed. A method may collect an identification of a source of a data record, a timestamp of the data record and a count of a number of records a data source has added to the stream with that timestamp, generate a unique identification with the identification of the source of the data record, the timestamp of the data record and the count of the number of records the data source has added to the stream with that timestamp, apply the unique identification to the data record and transmit the data record with the unique identification to a downstream operator within a data stream processing system.

Unique identification generation for records in a data streaming processing system

A system for providing continuous monitoring of data quality in a dynamic feed environment is disclosed. In particular, the system utilizes a feed inspection tool to detect anomalies in data gathering detected from feed metadata and anomalies in data measurement detected based on file contents. In order to do so, the feed inspection tool may aggregate, for a plurality of aggregation intervals, data feeds and associated metadata feeds. Once the data feeds and metadata feeds are aggregated, the feed inspection tool may generate, for a baseline model feed, baseline statistical models by utilizing historical data of the aggregated feeds in sliding windows of different lengths. The feed inspection tool may then identify, for a plurality of monitoring time delays, data outliers by comparing the aggregated feeds with the baseline model feed. A data quality feed based on the data outliers identified may then be generated and published.

System for continuous monitoring of data quality in a dynamic feed environment

A method, computer-readable medium, and device for processing a stream of records are disclosed. A method may receive a registration request from a data source to join in providing a plurality of records to the stream, synchronize a timing system with the data source, receive an initial timestamp from the data source, and post the initial timestamp to a plurality of operators or a plurality of subscribers before the data source is allowed to send records into the stream.

Network data source time management for data streaming processing system

A method includes parsing a regular pathway expression into fragments including an anchored fragment and at least one other fragment. A number of the fragments is based on at least a length limitation of the regular pathway expression. The method includes generating an operator directed acyclic graph (DAG) including non-operator nodes, operator nodes, and a root based on at least the anchored fragment. The method includes removing, from the operator DAG, at least one of the non-operator nodes and connecting a first operator node to a second operator node of the operator nodes. The first operator node includes an edge into the at least one removed non-operator node, and the second operator node includes an edge from the at least one removed node. The method includes executing the operator DAG on a graph database to return a pathway set comprising at least one pathway that satisfies the regular pathway expression.

Database evaluation of anchored length-limited path expressions

A method includes receiving, via a processor, a query comprising pathway variables and at least one evaluation time and determining an anchor set based on at least one of the pathway variables. The method also includes translating the pathway variables into a pathway algebraic expression based on the anchor set and the at least one evaluation time, and executing the pathway algebraic expression on a graph database to return a pathway set.

Time-based querying of graph databases

A system may include a processor, a user input, and memory comprising a graph database and executable instructions. The executable instructions may cause the processor to effectuate operations. The operations include receiving, via the user input, a query comprising a class generalization and pathway variables. The operations include identifying a query class based on at least the class generalization and determining an anchor set based on at least one of the pathway variables. The operations also include translating the pathway variables into a pathway algebraic expression based on the anchor set and the query class and executing the pathway algebraic expression on the graph database to return a pathway set. The class generalization comprises at least one of a component, a class field, or metadata.

Concept based querying of graph databases

A method, computer-readable medium, and device for processing a stream of records with a guarantee that each record is accounted for exactly once are disclosed. A method may receive, via a first operator, a data stream having a plurality of records, the plurality of records provided by a plurality of first data sources; allocate the data stream to a plurality of shards of the first operator; process the plurality of records by each shard of the plurality of shards to generate a first output stream, where each shard being implemented with at least two replicas; and output the first output stream to a third operator or a subscriber.

Network stream processing to ensuring a guarantee that each record is accounted for exactly once

A system may include a processor, an input, and memory comprising a graph database and executable instructions. The executable instructions may cause the processor to effectuate operations. The operations may include receiving, via the input, a query comprising pathway variables. The operations may also include determining an anchor set based on at least one of the pathway variables and identifying an evaluation order based on the anchor set. The operations may also include translating the pathway variables into a pathway algebraic expression based on the evaluation order. The operations may include executing the pathway algebraic expression on the graph database to return a pathway set.

Systems and methods for modeling networks

A device, method and computer-readable medium for recovering a replica in an operator in a data streaming processing system are disclosed. A method may obtain a checkpoint in an input data stream, determine a maximum-timestamp at the checkpoint in the input data stream, calculate a completeness point that is greater than the maximum-timestamp for an output data stream and process data records from the checkpoint onwards that have a respective timestamp that is greater than or equal to the completeness point that was calculated to generate a new replica to replace a failed replica.

Recovering a replica in an operator in a data streaming processing system

A method and system for providing query aware partitioning are disclosed. For example, the method receives a query plan comprising a plurality of queries, and classifies each one of the plurality of queries. The method computes an optimal partition set for each one of the plurality of queries, and reconciles the optimal partition set of each one of the plurality of queries with at least one subset of queries of the plurality of queries. The method selects at least one reconciled optimal partition set to be used by each query of the plurality of queries, and stores the selected at least one reconciled optimal partition set in a computer readable medium.

Growing community of inventors

New York, NY, United States of America

Vladislav Shkapenyuk