International Business Machines Corporation

Hong Min

Bharat Sukhwani

Sameh W Asaad

Mathew S Thoennes

Parijat Dube

Gong Su

Lei Yu

Xiao Li

Serge Bourbonnais

Hubertus Franke

Yuk Lung Chan

Fan Jing Meng

Jia Qi Li

Tian Wu

Yefim Shuf

Terence Patrick Purcell

Balakrishna Raghavendra Iyer

You-Chin Gene Fuh

Akshar Kaul

Manish Kesarwani

Jonathan W Wierenga

Yat On Lau

Lin Y Yang

Austin F M D'Costa

Christian Zentgraf

Petr Novotny

Donna N Dillenberger

Ke Wei Wei

Qi Zhang

Donald W Schmidt

Harold Wade Cain, Iii

Sameep Mehta

Guogen Zhang

Michel H T Hack

James Zu-Chia Teng

Bernard V Brezzo

Donna N Eng Dillenberger

Gagandeep Singh

Kan Zhang

Miao Zheng

Yi Jin

Allan Benjamin Lebovitz

Tian Tian

Zhi Shuai Han

Xi Long Zheng

Ou Jin

Xiaobo Wang

Ravi Nair

Elpida Tzortzatos

Roman Vaculin

Scott Brady Compton

Ulrich Alfons Finkler

Jing Yan Ma

Shyam Ramji

Motohiro Kawahito

Takuya Nakaike

Jie Xu

Venkat K Balagurusamy

Peter Bendel

Fen-Ling Lin

James Willis Pickel

Li Cao

Thanh Lam Hoang

Heng Liu

Nan Jiang

Patrick Dooling Bossman

Jeffrey Allen Berger

Qi Liang

Brian Thinh-Vinh Tran

Purvi Sharadchandra Patel

Chi Liu

Christopher John Crone

Eysha S Powers

Shaikh Shahriar Quader

Ze Ming Zhao

Paul J Wanish

Cecilia Carranza Lewis

Shuang Ys Yu

Weixiong Rao

Chunfeng Pei

Meichi M Lin

Dongsheng Chen

Akiko Hoshikawa

Yue Huang

Xiang Wei Zhou

Kanthi Sarpatwar

Matthew S Thoennes

Zhongliang Zhang

Kou-Horng A Yang

Egide Van Aerschot

Abhishek Dokania

Peng Zhao

Sahil Dureja

Gayathiri Chandran

Zhen Hua Dong

En Zhong Wang

Sameh W Assad

In accordance with an embodiment of the invention, a method is provided for performing data pre-processing operations during data preparation of a machine learning lifecycle. The method includes defining one or more data pre-processing functions for applying to data stored in a dataset, executing one or more learn functions for learning the data, and executing one or more transform functions for transforming the data. Each of the one or more learn functions generates a first Structured Query Language (SQL) statement representing a definition of corresponding learn function for corresponding defined data pre-processing function. Each of the one or more transform functions generates a second SQL statement representing a definition of corresponding transform function for corresponding defined data pre-processing function. The dataset is stored in a database.

Performing data pre-processing operations during data preparation of a machine learning lifecycle

A technique of data classification and data protection includes recording, in a lineage repository, dependence relationships between data objects and recording, in a classification repository, data classifications applicable to data objects. The data classifications each indicate a respective class of sensitive data to which data protection is to be applied. Based on receiving a classification job requesting classification of a first data object, a processor determines whether the lineage repository records a dependence relationship with a second data object. Based on determining that the lineage repository records a dependence relationship with the second data object, the processor associates in the classification repository at least a particular data classification applicable to the second data object with the first data object. The processor applies data protection applicable to the particular data classification to the first data object. In some cases, the processor may order application of classification models based on cost metrics.

Data protection utilizing efficient data classification

A method, apparatus and computer program product that provides multi-phase privacy-preserving inferencing in a high throughput data environment, e.g., to facilitate fraud prediction, detection and prevention. In one embodiment, two (2) machine learning models are used, a first model that is trained in the clear on first transaction data, and a second model that is trained in the clear but on the first transaction data, and user data. The first model is used to perform inferencing in the clear on the high throughput received data. In this manner, the first model provides a first level evaluation of whether a particular transaction might be fraudulent. If a transaction is flagged in this first phase, a second more secure inference is then carried out using the second model. The inferencing performed by the second model is done on homomorphically encrypted data. Thus, only those transactions marked by the first model are passed to the second model for secure evaluation.

Multi-phase privacy-preserving inferencing in a high volume data environment

An approach is provided in which a method, system, and program product analyze, while training a machine learning model, a set of first data transformation operators in a first data preparation pipeline that generates a plurality of constructed features from a set of training data. The method, system, and program product create a plurality of second data preparation pipelines from the first data preparation pipeline, wherein the set of first data transformation operators are converted to a set of second data transformation operators and each assigned to one of the plurality of second data preparation pipelines. The method, system, and program product deploy the plurality of second data preparation pipelines to a runtime system.

Auto feature preparation for high performance online inferencing

Embodiments of the present invention provide methods, computer program products, and systems. Embodiments of the present invention can receive, by a computing device, a request to access a datapoint of a machine learning dataset contained in a database. Embodiments of the present invention can access, by the computing device, a virtual data frame that includes a schema which represents a structure of the machine learning dataset in the database. Embodiments of the present invention can retrieve, by the computing device, the datapoint of the machine learning utilizing the virtual data frame and return, by the computing device, the retrieved datapoint in response to the request.

Processing large machine learning datasets

An approach to optimize performance for large scale inference models. Data in the form of images is received from sensors such as cameras. The data is processed to generate data tags associated with the context of the image and portion the images. Model tags are generated based on data characteristics or user input. The tags and their associated data are added to a time-based queue for delivery to the appropriate inference models. Based on the embedded delivery time and frequency, the portioned images are delivered to the appropriate inference models.

Inference model optimization

A computer-implemented method, a computer system, and a computer program product for trajectory similarity search is provided. The present invention may include, in response to receiving, by one or more processors, a search request for at least one trajectory similar to a query trajectory, determining, by one or more processors, a respective similarity between a query trajectory and a plurality of trajectories by calculating, in a synchronized way, a spatial distance measure and a time difference measure between the query trajectory and the plurality of trajectories. The present invention may further include, identifying, by one or more processors, the at least one trajectory from the plurality of trajectories based on the respective similarity between the query trajectory and the plurality of trajectories.

Trajectory similarity search

Embodiments of the invention are directed to annotating a log based on processing log documentation. Aspects include obtaining the data set and obtaining a set of templates, wherein one or more template of the set of templates includes a variable labeled as the data type. Aspects also include annotating the data set based on the set of templates, wherein the annotating includes labeling each value of the variable in the data set with the data type. Aspects further include creating an updated data set by replacing each value of the variable in the data set that is labeled with the data type with a new data value.

Removing data having a data type from a data set

Aspects of the invention include computer systems, computer-implemented methods, and computer program products configured to perform message correlation extraction for mainframe operation. A non-limiting example computer-implemented method includes receiving a first message log entry having a message identifier and message field data. The first message log entry is pre-processed to determine the message identifier and to tokenize the message field data. A second message log entry is identified based on at least one of the message identifier and the tokenized message field data. The method further includes determining that the second message log entry is correlated with the first message log entry and providing an output comprising the message correlation between the first message log entry and the second message log entry.

Message correlation extraction for mainframe operation

Embodiments of the invention are directed to annotating a log based on processing log documentation. Aspects include obtaining the log having a plurality of entries. Aspects also include creating a set of log entry templates by processing the log documentation associated with the log, wherein each log entry template includes one or more constants and one or more variables. Aspects further include annotating each of the plurality of entries based on the set of templates, wherein the annotating includes labeling each value of the one or more variables with a variable name.

Annotating a log based on log documentation

According to an aspect, a method includes searching for a correlated log identifier in a correlation database based on detecting a metrics-based anomaly. The method also includes providing, in a problem diagnosis, related log information associated with the correlated log identifier based on locating one or more log entries including the correlated log identifier in a same time window as the metrics-based anomaly. The method further includes searching for a correlated metric in the correlation database based on detecting a log-based anomaly and providing, in the problem diagnosis, related metric information associated with the correlated metric based on locating one or more metrics records including the correlated metric in the same time window as the log-based anomaly.

Correlation-based multi-source problem diagnosis

Methods, systems, and computer program products for identifying and redacting data from diagnostic operations via monitoring of data are provided herein. A computer-implemented method includes automatically monitoring data, which includes determining which portions of the data are accessed during a diagnostic operation; parsing an input file into portions of parsed data, wherein the input file comprises an initial output of the diagnostic operation; classifying the portions of parsed data into classes by applying at least one of multiple classification models to the parsed data, wherein the at least one classification model is specific to the accessed portions of data; automatically identifying sensitive data in the classified portions of parsed data by applying a class-to-sensitivity mapping technique to the classified portions of parsed data; redacting the identified sensitive data from the input file; and generating and outputting an updated output of the diagnostic operation based on the redacting.

Dynamically identifying and redacting data from diagnostic operations via runtime monitoring of data sources

Techniques for refinement of data pipelines are provided. An original file of serialized objects is received, and an original pipeline comprising a plurality of transformations is identified based on the original file. A first computing cost is determined for a first transformation of the plurality of transformations. The first transformation is modified using a predefined optimization, and a second cost of the modified first transformation is determined. Upon determining that the second cost is lower than the first cost, the first transformation is replaced, in the original pipeline, with the optimized first transformation.

Streamlining data processing optimizations for machine learning workloads

Techniques include collecting current logs from distributed sources, selecting a group of the current logs that are from a related source of the distributed sources, and generating a feature vector using the group of the current logs. A current status model is created for the feature vector using the group of the current logs. One or more anomalies are determined in the group of the current logs based on a difference between the current status model and a reference status model, the reference status model being based on history logs.

Log-based status modeling and problem diagnosis for distributed applications

One embodiment provides a method, including: receiving, from a user, a dataset for encryption before its storage at a data storage location, wherein the dataset comprises a plurality of portions; identifying (i) attributes of the dataset and (ii) dataset dependencies; generating a recommendation for an encryption scheme to be used for the dataset, wherein the generating comprises (i) generating, based upon the attributes and the dataset dependencies, a recommendation of an encryption scheme for each portion of the dataset and (ii) identifying, based upon the dataset dependencies, a key label for each portion of the dataset, wherein the key label identified for a portion of the dataset that is dependent on another portion of the dataset is the same as the key label identified for said another portion of the dataset; and providing, to the user, (i) the generated recommendation and (ii) a description identifying reasons for the generated recommendation.

Encryption scheme recommendation

Growing community of inventors

Poughkeepsie, NY, United States of America

Hong Min