International Business Machines Corporation

Hewlett-packard Development Company, L.p.

Opera Solutions Usa, LLC

Amir Bar-Or

Michael James Beckerle

Nitzan Peleg

Jean-Manuel Van Thong

Tal Drory

Leonidas Kontothanassis

David Goddeau

Yuansong Liao

Manish Anand Bhide

Sriram Kolinjivadi Padmanabhan

Manoj Kumar Agarwal

David Konopnicki

Christian Hubmann

Ding Huang

Thejaswi Raje Gowda

Sebastian Ertel

Yan Ge

Laks Srinivasan

Code and data versioning allows developers to work on code and data without affecting production code and data and without affecting the development activities of other developers. Code and data being worked on by a developer are associated with a task. The system automatically determines the dataset to use for a given development task from among a production dataset, a latest dataset, or a temporary dataset associated with the development task so that development code does not have to be modified to read from a specific dataset.

System and method for code and data versioning in computerized data modeling and analysis

A system and method for rapid development and deployment of reusable analytic code for use in computerized data modeling and analysis is provided. The system includes a centralized, continually updated environment to capture pre-processing steps used in analyzing big data, such that the complex transformations and calculations become continually fresh and accessible to those investigating business opportunities. The system incorporates deep domain expertise as well as ongoing expertise in data science, big data architecture, and data management processes. In particular, the system allows for rapid development and deployment of analytic code that can easily be re-used in various data analytics applications, and on multiple computer systems.

System and method for rapid development and deployment of reusable analytic code for use in computerized data modeling and analysis

An interactive reporting tool allows the user to define, modify, and selectively execute a sequence of queries in an interactive manner. A sequence of queries is entered into rows of an interactive query code table. When the user selects a given row i or a cell in a given row i, then the data table is updated to show the results of the queries 1 through i, such that the user effectively can step through the queries in any order (i.e., sequential or non-sequential, forward or backward) to see the results of each step.

System and method for interactive reporting in computerized data modeling and analysis

A sequence of queries is divided into stages, where each stage involves one pass over the data, such that the sequence of queries can be executed using the minimum number of passes over the data. The sequence of queries may be processed into a functional dependency graph that represents the relationships between query parameters (signals) and query operations, and the functional dependency graph then may be processed to divide the queries into a number of successive stages such that each stage includes queries that can be executed based on data that exists prior to execution of that stage. A sequence of queries may, and often does, require that one or more intermediate values or datasets be generated using an aggregate function.

System and method for optimized query execution in computerized data modeling and analysis

A method, a computer program product and a system identify partition locations within an extended markup language (XML) document without parsing so as to process portions of said document in parallel. The XML document includes sections required to remain continuous. The document is scanned for continuous sections without parsing, and boundaries of the initial partitions are adjusted to reside outside the continuous sections to determine resulting partitions for the document. The resulting partitions may be processed in parallel to provide the document information for storage.

Finding partition boundaries for parallel processing of markup language documents

Provided are techniques for pipeline optimization based on polymorphic schema knowledge. A hierarchical document to be processed by a pipeline of transformations is received. It is determined whether a next downstream transformation accesses content of each schema node in an associated input schema, wherein the input schema is a polymorphic schema. In response to determining that the next downstream transformation is not accessing the content of each schema node in the associated input schema, data items corresponding to each schema node that the next downstream transformation is not accessing are collected into a single compressed event. The collected items are passed to the next downstream transformation as the single compressed event.

Pipeline optimization based on polymorphic schema knowledge

Provided are techniques for processing data items. A limit on the number of dequeue operations allowed in a current step of processing for a queue-like data structure is set, wherein the number of allowed dequeue operations limit at least one of an amount of CPU resources and an amount of memory resources to be used by an operator. The operator to perform processing is selected and the operator is activated by passing control to the operator, which then dequeues data constrained by the limits set. In response to receiving control back from the operator, the data structure size is examined to determine whether the operator made forward progress in that the operator enqueued or dequeued at least one data item.

Asynchronous data structure pull application programming interface (API) for stream systems

A disclosed embodiment relates to a system for populating an index table. The system comprises a component that decomposes a spatial object into a set of object pairs, a component that employs a value associated with the object pairs to sort the set of object pairs into a sequence of object pairs, a component that uses a sliding border analysis to create a subset of the sequence of object pairs depending upon a position of the object pair in the sequence of object pairs, and a component that populates the index table with the subset of object pairs.

Method and apparatus for populating an index table

The disclosed embodiments relate to an apparatus for performing query operations. The apparatus comprises a base table having spatial objects, an index table having data entries that are associated with the spatial objects. A module that is adapted to perform the query operation is configured to convert a query window into values, create a scan range for each of the values with a begin range value, an end range value, and a stop condition, scan the data entries for each of the scan range to identify one of the end range value and the stop condition, and return a result based upon the data entries that are within the scan range for each of the values.

Method and apparatus for querying spatial data

Provided are techniques for compilation of hierarchical data processing. A data flow diagram including one or more operators, wherein each operator includes at least one of an incoming arc and an outgoing arc, is received. For each operator, for each incoming arc, it is validated that an arc input formal schema is compatible with a schema rooted in a context node in an arc input actual schema, and, for each outgoing arc, an arc output formal schema is computed based on operator logic and operator inputs and an arc output actual schema is computed from the arc input actual schema by replacing the context node of the arc input actual schema with the arc output formal schema.

Compilation model for processing hierarchical data in stream systems

Provided are techniques for optimizing the processing of hierarchical data. A linear processing graph is received, wherein the linear processing graph includes a plurality of operators, wherein each operator in the plurality is connected to at least one other operator by an arc, wherein hierarchical data flows on arcs, wherein the operators read and replace identified subregions within the hierarchical data flowing into the operators on the arcs, and wherein the operators do not modify the hierarchical data outside of these identified subregions. For each operator in the linear processing graph, a minimal set of dependent upstream operators on which that operator depends is found by examining how the identified subregions are created in the linear processing graph through obtaining a set of operators on which that operator depends, by analyzing dependencies carried by a set of vector nodes of the hierarchical data in an input schema of the operator, and, for each of the vector nodes, by analyzing an associated set of scalar nodes, wherein finding the minimum set of operators includes taking into consideration data preservation characteristics of the plurality of operators and taking into consideration structural-order preservation characteristics of the plurality of operators. The linear processing graph is rewritten to create a new graph that expresses dependencies based on the minimal set of dependent upstream operators for each operator.

Optimization model for processing hierarchical data in stream systems

A method for shedding signal load in a network having a plurality of devices is disclosed herein. The method comprises computer-implemented steps of (a) issuing a load-shedding policy, where the policy includes (i) fraction f of the signal to be discarded; (ii) device D where fraction f of the signal is to be discarded; and (iii) pattern P of discarding fraction f of the signal; (b) transmitting the issued load-shedding policy to device D; and (c) discarding portions of the signal at device D according to pattern P until fraction f of the signal is discarded.

Method and system for shedding load in physiological signals processing

A system and method for implementing a unified model for integration systems is presented. A user provides inputs to an integrated language engine for placing operator components and arc components onto a dataflow diagram. Operator components include data ports for expressing data flow, and also include meta-ports for expressing control flow. Arc components connect operator components together for data and control information to flow between the operator components. The dataflow diagram is a directed acyclic graph that expresses an application without including artificial boundaries during the application design process. Once the integrated language engine generates the dataflow diagram, the integrated language engine compiles the dataflow diagram to generated application code.

System and method for simulating data flow using dataflow computing system

A method of load-shedding in a system having a plurality of signal sources is disclosed. The method comprises the steps of: (a) for each signal source Sselected from a set of sources {S, S, . . . }, each source having state pselected from a set of states {p, p, . . . , p; p, p, . . . , p; . . . }, wherein i is an integer greater than one, N is an integer not less than two and k is an integer from 1 to N: (i) obtaining fractions f, wherein each 0&#x2266;f&#x3c;1 is a fraction of a signal from source Sto be discarded if source Sis in state p; and (ii) determining respective state pof signal source Sbased on the signal from source S; and (b) for each source signal Sin determined state p, discarding fraction fof a signal from source S.

Growing community of inventors

Newton, MA, United States of America

Amir Bar-Or