International Business Machines Corporation

Sap Ag

Eitan D Farchi

Orna Raz

Aviad Zlotnick

Shmuel Ur

Shachar Fienblit

Rachel Tzoref-Brill

Orna Raz-Pelleg

Ramani Ranjan Routray

Oded Margalit

Howard M Hess

Hana Chockler

Omer Y Boehm

Sergey Novikov

Yousef Shajrawi

Michael Vinov

Moran Shochat

Sheng Hua Bao

Itai Segall

Marcel Zalmanovici

Amiram Hayardeny

Sheli Rahav

Dalit Tzafrir

Marina Biberstein

Pathirage D Perera

Benyamin Godlin

Rivka Mayraz Matosevich

Diptikalyan Saha

Ryan Thomas Rawlins

David Carmel

Dale E Blue

Abel Alkon Gordon

Lawrence Carter Blount

Kenneth Nagin

Ziv Glazberg

Onn Menahem Shehory

Alan Hartman

Ifat Nuriel

Yair Allouche

Nadav Yosef Har'el

Rachel Brill

Paul Regis Kram

Aviad Cohen

Eliran Roffe

Pathirage Dinindu Sujan Udayanga Perera

Michael Edward Factor

Parijat Dube

Yoelle Smadja Maarek

Tamer Salman

Andrew C Hicks

Michael E Gildein

Avi Ziv

Doron Cohen

Ishai Rabinovitz

Andre Heilper

Michael Herscovici

Aya Soffer

Ayman Jarrous

Ronald Fagin

Ofer Haim Biller

Daniel Geist

Rachel Tzoref

Maayan Goldstein

Sweta Singh

Ravid Sagy

Vaibhav Murlidhar Kulkarni

Yael Shaham-Gafni

Yarden Nir-Buchbinder

Shay Bushinsky

Yoram Simha Adler

Ehud Trainin

Richard D Prewitt, Jr

Yael Petruschka

Pathirage D S U Perera

Vivek Krishnamurthy

Nakul Chakrapani

Aharon Brodie

Sandeep Hans

Devika Sondhi

Mario Dominic Savio Briggs

Deepak Anil Mahajan

Gil Ratsaby

Shmuol Ur

George Inness

Yarden Nir

Shay Gammer

Yochai Ben-Chaim

Itay Segall

Techniques for testing and fault detection are disclosed. These techniques include generating a set of test cases for a system under test (SUT), the set of test cases based on attribute-value pairs modeled as input to the SUT. The techniques further include augmenting the set of test cases. This includes locating a missing counterpart for a first combination of values in a first test case in the set of test cases, based on identifying a number of instances of the first combination of values in the set of test cases, generating a new test case based on modifying the first test case to act as the missing counterpart, and adding the new test case to the set of test cases. The techniques further include identifying a fault for the SUT based on executing the augmented set of test cases.

Self diagnosing test suite using a double pair-wise combinatorial test solution

An example system includes a processor to receive a set of features, a set of relations between the features, and a set of target features. Each of the target features is associated with a number of the relations. The processor can generate a hypergraph based on the features and the relations. The processor also can select a subset of features based on a transitive closure of the hypergraph for each of the target features. The processor can transmit the selected subset of features.

Feature selection using hypergraphs

A method including: receiving a set of data representing usage by entities of objects in a computing resource; extracting, from the initial set of data, one or more feature vectors representing the usage by one of the entities with respect to the objects; generating, from the feature vectors, a feature matrix; with respect to each entry in the feature matrix: (i) assigning a binary value to the entry, based on a predefined usage threshold, (ii) identifying, among the one or more entities, k nearest neighbor entities with respect to the one of the entities, based on a predefined distance threshold, and (iii) modifying the usage value of the entry, based on usage values associated with each of the k nearest neighbor entities with respect to the one of the objects; and updating the feature matrix with the modified usage values, to obtain a manipulated feature matrix.

Anomaly detection of entity behavior

In some examples, a system for decorating network traffic flows with outlier scores includes a processor and a memory device to store traffic flows received from a network. The processor is configured to receive a set of traffic flows from the memory device and generate a tree model to split the traffic flows into clusters of traffic flows. Each cluster corresponds with a leaf of the tree model. The processor is further configured to generate machine learning models for each of the clusters of traffic flows separately. For a new traffic flow, the processor is configured to identify a specific one of the machine learning models that corresponds with the new traffic flow, compute an outlier score for the new traffic flow using the identified specific one of the machine learning models, and decorate the new traffic flow with the outlier score.

Cluster-based outlier scoring of network traffic

Embodiments are for generating a digital signature of a query execution plan using similarity hashing. A technique includes generating a node digital signature for nodes in a query and generating an edge digital signature for edges in the query, the edges connecting the nodes. The technique includes selecting at least one previously executed query based on the node digital signature and the edge digital signature for the query and causing the query to be processed according to an assignment associated with the at least one previously executed query.

Generate digital signature of a query execution plan using similarity hashing

Methods, systems, and computer program products for path-coverage directed black box application programming interface (API) testing are provided herein. A computer-implemented method includes determining constraints based on inputs and corresponding outputs of an API in a production environment; generating initial test inputs based at least in part on the constraints; creating a program dependency graph based on trace sequences and request-response data obtained in response to providing the initial test inputs to an endpoint of the API; enhancing the program dependency graph by generating additional test inputs directed to one or more paths of the dependency graph; identifying, based on the enhanced program dependency graph, at least a portion of the API that is not covered by an existing test suite; and using the enhanced program dependency graph to generate new test cases for the test suite based on the identifying.

Path-coverage directed black box API testing

A method, apparatus and a product for determining a performance measurement of predictors. The method comprises obtaining a dataset comprising data instances. Each data instance is associated with a label; obtaining a predictor. The predictor is configured to provide a prediction of a label for a data instance; determining a plurality of data slices that are subsets of the dataset. computing, for each data slice in the plurality of data slices and based on an application of the predictor on each data instance that is mapped to the data slice, a performance measurement that is indicative of a successful label prediction for a data instance comprised by the data slice, whereby obtaining a plurality of performance measurements; based on the plurality of performance measurements, computing a performance measurement of the predictor over the dataset; if the performance measurement of the predictor is below a threshold, performing a mitigating action.

Performance measurement of predictors

A method, system and computer program product, the method comprising: obtaining a first model trained upon cases and labels, the first model providing a prediction in response to an input case; obtaining a second model trained using the cases and indications whether a predictions of the first model are correct, the second model providing a correctness prediction for the first; determining a case for which the second model predicts that the first provides an incorrect prediction; further training the first model also on a first corpus including the case and a label, thereby improving performance of the first model; providing the case to the first model to obtain a first prediction; and further training the second model also on a second corpus including the case and a correctness label, the correctness label being 'correct' if the first prediction is equal to the label, thereby improving performance of the second model.

Labeling a dataset

A mechanism is provided in a data processing system having a processor and a memory. The memory comprises instructions which are executed by the processor to cause the processor to implement a training system for finding an optimal surface for hierarchical classification task on an ontology. The training system receives a training data set and a hierarchical classification ontology data structure. The training system generates a neural network architecture based on the training data set and the hierarchical classification ontology data structure. The neural network architecture comprises an indicative layer, a parent tier (PT) output and a lower leaf tier (LLT) output. The training system trains the neural network architecture to classify the training data set to leaf nodes at the LLT output and parent nodes at the PT output. The indicative layer in the neural network architecture determines a surface that passes through each path from a root to a leaf node in the hierarchical ontology data structure. The training system trains a classifier model for a cognitive system using the surface and the training data set.

Optimizing hierarchical classification with adaptive node collapses

A method, apparatus and product for identifying data drifts. The method comprising: obtaining a baseline dataset of instances in a feature space, each of wherein being associated with a label; determining a set of clusters in the feature space, based on the baseline dataset; determining a baseline distribution of instances over the set of clusters based on the baseline dataset; for each cluster, computing a performance metric for a predictor for the each cluster, wherein the predictor is configured to estimate an estimated label for an instance, wherein the performance metric is indicative of a successful estimation of the predictor to a portion of the baseline dataset that is comprised by the cluster; obtaining a second dataset, wherein the second dataset comprising instances in the feature space; determining a second distribution of instances over the set of clusters, wherein said determining the second distribution is based on the second dataset; and based on the second distribution and on the baseline distribution, and based on at least one performance metric of at least one the cluster of the set of clusters, identifying a data drift in the second dataset with respect to the baseline dataset.

Identifying data drifts that have an adverse effect on predictors

A method, computer system, and a computer program product for assessing a likelihood of success associated with developing at least one machine learning (ML) solution is provided. The present invention may include generating a set of questions based on a set of raw training data. The present invention may also include computing a feasibility score based on an answer corresponding with each question from the generated set of questions. The present invention may then include, in response to determining that the computed feasibility score satisfies a threshold, computing a level of effort associated with developing the at least one ML solution to address a problem. The present invention may further include presenting, to a user, a plurality of results associated with assessing the likelihood of success of the at least one ML solution.

Estimating feasibility and effort for a machine learning solution

A method, system and computer program product, the method comprising: obtaining computer code of an employed system comprising a plurality of components; obtaining data related to operating the plurality of components; based on the computer code and the data, identifying: a first component from the plurality of components, to be maintained; and a second component from the plurality of components, to be at least partly replaced by a machine learning component; and providing to a user an identification of the first component and the second component.

Method and apparatus for employing machine learning solutions

A method, apparatus and a computer program product for automated data slicing based on an Artificial Neural Network (ANN). The method comprising: obtaining an ANN, wherein the ANN is configured to provide a prediction for a data instance, wherein the ANN comprises a set of nodes having interconnections therebetween; determining an attribute vector based on a subset of the nodes of the ANN; determining, based on the attribute vector, a plurality of data slices; obtaining a testing dataset comprising testing data instances; computing, for each data slice, a performance measurement of the ANN over the data slice, wherein said computing is based on an application of the ANN on each testing data instance that is mapped to the data slice; and performing an action based on at least a portion of the performance measurements of the data slices.

Automated data slicing based on an artificial neural network

A computer system trains a machine learning model. A vector representation is generated for each document in a collection of documents. The documents are clustered based on the vector representations of the documents to produce a plurality of clusters. A training set is produced by selecting one or more documents from each cluster, wherein the selected documents represent a sample of the collection of documents to train the machine learning model. The machine learning model is trained by applying the training set to the machine learning model. Embodiments of the present invention further include a method and program product for training a machine learning model in substantially the same manner described above.

Generating training sets to train machine learning models

A method and a computer program product for identification and improvement of machine learning (ML) under-performance The method comprises slicing data of ML model based on a functional model representing requirements of a system utilizing the ML model. The functional model comprises a set of attributes and respective domain of values. Each data slice is associated with a different valuation of one or more attributes of the functional model. Each data instance of the ML model is mapped to one or more data slices, based on valuation of the attributes. A performance measurement of the ML model over is computed for each data slice, based on an application of the ML model on each data instance that is mapped to the data slice. A Determination whether ML model adheres to a target performance requirement may be performed based on the performance measurements of the data slices.

Data slicing for machine learning performance testing and improvement

Growing community of inventors

Parades Hana, Israel

Eitan D Farchi