International Business Machines Corporation

Sergey Krutyolkin

Uri Shani

Aviad Zlotnick

Ariel Landau

Dagan Gilat

David Harold Berk

Joel Arditi

Aviad Sela

Ronen Shevach

Doug Billings

A data processing method is provided. The method comprises collecting a stream of data records received from one or more data sources connected in a communications network; dividing the stream of data records into sets of data records for parallel processing by a plurality of concurrently running tasks, wherein a first task loads a persistent index associated with a first set of data records into memory to generate an in-memory version of the first persistent index for the first set of data records; and identifying duplicate and non-duplicate data records in the first set of data records, based on searching the in-memory version of the first persistent index.

Duplicate filtering in a data processing environment

A method of scheduling highly parallel jobs with global interdependencies is provided herein. The method includes the following steps: grouping input elements, each group being associated with an interdependency tag reflecting a level of interdependency between data associated with different input elements within a group; clustering the groups into collections of groups, wherein the clustered groups are associated with an interdependency tag reflecting a level of interdependency between groups, above a specified value; applying a conflict check to the collections of groups and to active jobs of a working set, to yield a conflict level between each collection of groups and each active job, by analyzing the interdependency tags of the collections of groups vis &#xe0; vis interdependency tags associated with the active jobs; and adding collections of groups into the working set, wherein added collections of groups are associated with a conflict level below an acceptable conflict level.

Scheduling highly parallel jobs having global interdependencies

A data processing method is provided. The method comprises collecting a stream of data records from one or more devices in a network; loading one or more persistent indexes associated with the stream of data records into memory; identifying duplicate data records in the stream of data records using the in-memory indexes; and updating a repository such that the duplicate data records are not stored in the repository or managed differently than non-duplicate data records.

A method for processing a document that includes a plurality of fields having respective contents. The method includes providing labels to be assigned respectively to the fields and one or more rules applicable to the contents of the fields according to the labels assigned thereto. A computer reads the contents of the fields, and assigns the labels to the fields responsive to application of the rules to the contents.

Automatic assignment of field labels

A method for processing an input image includes applying one or more transformations to the input image, whereby different shifts are applied to different pixels in the input image, so as to generate an output image. A region containing content of interest is selected in the output image, and the region corresponding to the selected region in the output image is located in the input image. The content of the located region in the input image is substituted for the content of the selected region in the output image.

Growing community of inventors

Haifa, Israel

Sergey Krutyolkin