International Business Machines Corporation

Shaikh Shahriar Quader

Nitin Kumar Gupta

Petr Novotny

Pedro Miguel Barbas

Hima Patel

Piotr Mierzejewski

Adrian Mahjour

Mona Nashaat Ali Elmowafy

Hong Min

David C Kalmuk

Darrell Christopher Reimer

Horst Cornelius Samulowitz

Marin Litoiu

Emmanouil Papangelis

Aindrila Basak

Nicolas Andres Jaramillo Duran

Sumona Mukhopadhyay

Chad Marston

A cleansing operation defined for a data structure of a database managed by a database management system is obtained. The cleansing operation is performed on data of the data structure to obtain clean data. The cleansing operation that is defined for the data structure and performed on data of the data structure is performed by the database management system.

In-database data cleansing

A cleansing operation is performed on data of a data structure to obtain clean data. The clean data is stored as part of the data structure; however, the clean data is independent of the data. A mapping is performed to provide a set of ordered data that includes the data and the clean data.

In-database data cleansing and independent store of clean data

Embodiments of the present invention provide methods, computer program products, and systems. Embodiments of the present invention can receive, by a computing device, a request to access a datapoint of a machine learning dataset contained in a database. Embodiments of the present invention can access, by the computing device, a virtual data frame that includes a schema which represents a structure of the machine learning dataset in the database. Embodiments of the present invention can retrieve, by the computing device, the datapoint of the machine learning utilizing the virtual data frame and return, by the computing device, the retrieved datapoint in response to the request.

Processing large machine learning datasets

Noisy labeled and unlabeled datapoint detection and rectification in a training dataset for machine-learning is facilitated by a processor(s) obtaining a training dataset for use in training a machine-learning model. The processor(s) applies ensemble machine-learning and a generative model to the training dataset to detect noisy labeled datapoints in the training dataset, and create a clean dataset with preliminary labels added for any unlabeled datapoints in the training dataset. Data-driven active learning and the clean dataset are used by the processor(s) to facilitate generating an active-learned dataset with true labels added for one or more selected datapoints of a datapoint pool including the detected noisy labeled datapoints and the unlabeled datapoints of the training dataset. The machine-learning model is trained by the processor(s) using, at least in part, the clean dataset and the active-learned dataset.

Data-analysis-based, noisy labeled and unlabeled datapoint detection and rectification for machine-learning

A quality determination method, system, and computer program product that includes performing a dimensionality reduction on a high-dimensional dataset to form a dimensional-reduced dataset and determining, using a machine learning tool executed on a computing device, a quality of the dimensional-reduced dataset via a review of an extracted feature extracted from the dimensional-reduced dataset.

Quality assessment of extracted features from high-dimensional machine learning datasets

Classification of erroneous cell data includes performing unsupervised pre-training of a machine learning model to learn a bidirectional encoder representation of data cells, obtaining an initial training set, with labeled training examples that correlate observed cell data to correct cell data, for training the machine learning model to classify cell data, automatically augmenting the initial training set to produce an augmented training set, where the augmenting includes identifying patterns in the labeled training examples, generating transformation functions, and using the transformation functions, learning an augmentation strategy and automatically generating additional training examples correlating erroneous data values to correct data values, and training the machine learning model using the augmented training set.

Classification of erroneous cell data

A DBMS training subsystem trains a DBMS workload-manager model with training data identifying resources used to execute previous DBMS data-access requests. The subsystem integrates each request's high-level features and compile-time operations into a vector and clusters similar vectors into templates. The requests are divided into workloads each represented by a training histogram that describes the distribution of templates associated with the workload and identifies the total amounts and types of resources consumed when executing the entire workload. The resulting knowledge is used to train the model to predict production resource requirements by: i) organizing production queries into candidate workloads; ii) deriving for each candidate a histogram similar in form and function to the training histograms; iii) using the newly derived histograms to predict each candidate's resource requirements; iv) selecting the candidate with the greatest resource requirements capable of being satisfied with available resources; and v) executing the selected workload.

Growing community of inventors

Scarborough, Canada

Shaikh Shahriar Quader