International Business Machines Corporation

Douglas Ronald Burdick

Berthold Reinwald

Xin Ru Wang

Shivakumar Vaithyanathan

Shirish Tatikonda

Yuanyuan Tian

Rajasekar Krishnamurthy

Prithviraj Sen

Matthias Boehm

Alexandre Valentinovich Evfimievski

Xinyi Zheng

Ioannis Katsis

Sebastian Schelter

Robert G Farrell

Yunyao Li

Mustafa Canim

Lucian Popa

Edwin Peter Dawson Pednault

Hima Prasad Karanam

Mauricio Antonio Hernandez-Sherrington

Vikas Sindhwani

Shajith I Mohamed

Marina Danilevsky Hailpern

Amol N Ghoting

Bogdan Alexe

Stefan Burnicki

Wei Cheng

Methods, systems, and computer program products for cooperative neural networks with spatial containment constraints are provided herein. A computer-implemented method includes dividing a processing task into multiple sub-tasks; training multiple independent neural networks, such that at least some of the multiple sub-tasks correspond to different ones of the multiple independent neural networks; defining, via implementing constraint-based domain knowledge related to the processing task in connection with the multiple independent neural networks, a constraint loss for a given one of the multiple sub-tasks, the constraint loss being dependent on output from at least one of the other multiple sub-tasks; and effecting re-training of at least a portion of the multiple independent neural networks by incorporating the constraint loss into at least one of the multiple independent neural networks.

Cooperative neural networks with spatial containment constraints

Methods, systems, and computer program products for vision-based cell structure recognition using hierarchical neural networks and cell boundaries to structure clustering are provided herein. A computer-implemented method includes detecting a style of the given table using at least one style classification model; selecting, based at least in part on the detected style, a cell detection model appropriate for the detected style; detecting cells within the given table using the selected cell detection model; and outputting, to at least one user, information pertaining to the detected cells comprising image coordinates of one or more bounding boxes associated with the detected cells.

Vision-based cell structure recognition using hierarchical neural networks and cell boundaries to structure clustering

A computer system, product, and method are provided to utilize machine learning to facilitate document processing. A document collection is introduced to an artificial neural network (ANN), which subjects the document collection to table region identification within discretized contiguous areas. The documents are assigned to one or more clusters responsive to the leveraged ANN. Documents are selectively evaluated from the clusters, and one or more label corrections are applied to the ANN. The ANN generates an updated document collection incorporating the applied one or more label corrections.

Interactive structure annotation with artificial intelligence

Organizing and/or aligning fragments of text that are included in a set of physical and/or digital documents so that the arrangement of the text fragments is in a readily understandable and meaningful format for a given reader. This organization and/or alignment uses a relation model of the various text fragments to correlate a meaning between and amongst the various text fragments to ultimately determine the final alignment and/or arrangement of those text fragments.

Organizing fragments of meaningful text

One embodiment provides a method, including: receiving two documents, one of the two documents having at least one table that includes the same information as a corresponding table in the other of the two documents, wherein (i) one of the two documents comprises the at least one table in an unstructured table representation and (ii) the other of the two documents comprises the at least one table in a structured table representation; identifying text elements within the at least one table in the unstructured table representation; matching the identified text elements with table elements within the at least one table in the structured table representation; and annotating the at least one table in the structured table representation based upon the matches between the table elements and text elements.

Automated non-native table representation annotation for machine-learning models

Vision-based cell structure recognition using hierarchical neural networks

Methods, systems, and computer program products for table recognition in PDF documents are provided herein. A computer-implemented method includes discretizing one or more contiguous areas of a PDF document; identifying one or more white-space separator lines within the one or more discretized contiguous areas of the PDF document; detecting one or more candidate table regions within the one or more discretized contiguous areas of the PDF document by clustering the one or more white-space separator lines into one or more grids; and outputting at least one of the candidate table regions as a finalized table in accordance with scores assigned to each of the one or more candidate table regions based on (i) border information and (ii) cell structure information.

Table recognition in portable document format documents

A computer-implemented method is provided that includes identifying an input dataset formatted as an input matrix, the input matrix including a plurality of rows and a plurality of columns. The computer-implemented method also includes dividing the input matrix into a plurality of input matrix blocks. Further, the computer-implemented method includes distributing the input matrix blocks to a plurality of different machines across a distributed filesystem, and sampling, by at least two of the different machines in parallel, at least two of the input matrix blocks. Finally, the computer-implemented method includes generating at least one sample matrix based on the sampling of the at least two of the input matrix blocks.

Single-pass distributed sampling from block-partitioned matrices

Parallel execution of machine learning programs is provided. Program code is received. The program code contains at least one parallel for statement having a plurality of iterations. A parallel execution plan is determined for the program code. According to the parallel execution plan, the plurality of iterations is partitioned into a plurality of tasks. Each task comprises at least one iteration. The iterations of each task are independent. Data required by the plurality of tasks is determined. An access pattern by the plurality of tasks of the data is determined. The data is partitioned based on the access pattern.

Hybrid parallelization strategies for machine learning programs on top of mapreduce

Described herein are methods, systems and computer program products for entity resolution. Entity resolution, also known as entity matching or record linkage, seeks to identify equivalent data objects between or among datasets. An example method includes creating a deterministic model by defining an entity to be resolved, selecting two datasets for comparison, defining matching predicates for attributes of the datasets to select a set of candidate matches, and defining a precedence rule for the candidate matches to select a subset of the candidate matches. The method includes running the deterministic model on the two datasets. Running the deterministic model includes applying the matching predicates and the precedence rule to data in the datasets that correspond to the attributes. The method also includes applying a cardinality rule to results of the running, and outputting the matching candidates for which the cardinality rule is satisfied.

Entity resolution between datasets

In a method for analyzing a large data set using a statistical computing environment language operation, a processor generates code from the statistical computing environment language operation that can be understood by a software system for processing machine learning algorithms in a MapReduce environment. A processor transfers the code to the software system for processing machine learning algorithms in a MapReduce environment. A processor invokes execution of the code with the software system for processing machine learning algorithms in a MapReduce environment.

R-language integration with a declarative machine learning language

Hybrid parallelization strategies for machine learning programs on top of MapReduce are provided. In one embodiment, a method of and computer program product for parallel execution of machine learning programs are provided. Program code is received. The program code contains at least one parallel for statement having a plurality of iterations. A parallel execution plan is determined for the program code. According to the parallel execution plan, the plurality of iterations is partitioned into a plurality of tasks. Each task comprises at least one iteration. The iterations of each task are independent.

Hybrid parallelization strategies for machine learning programs on top of MapReduce

Systems and methods for processing Machine Learning (ML) algorithms in a MapReduce environment are described. In one embodiment of a method, the method includes receiving a ML algorithm to be executed in the MapReduce environment. The method further includes parsing the ML algorithm into a plurality of statement blocks in a sequence, wherein each statement block comprises a plurality of basic operations (hops). The method also includes automatically determining an execution plan for each statement block, wherein at least one of the execution plans comprises one or more low-level operations (lops). The method further includes implementing the execution plans in the sequence of the plurality of the statement blocks.

Growing community of inventors

San Jose, CA, United States of America

Douglas Ronald Burdick