International Business Machines Corporation

Matthew R Arnold

Michael James Hind

Martin J Hirzel

Jiri Navratil

Stephen Fink

Peter Francis Sweeney

Michael Desmond

David Paul Grove

Benjamin Tyler Elder

Evelyn Duesterwald

Vadakkedathu Thomas Rajan

Darrell Christopher Reimer

Jerome Simeon

Avraham Ever Shinnar

Vijay Anand Saraswat

Eran Yahav

Martin Vechev

Jeffrey Scott Boston

Lionel A Villard

Sameep Mehta

Julian Timothy Dolby

Ramya Raghavendra

Kalapriya Kannan

David Francis Bacon

Kush Raj Varshney

Erik Richter Altman

John Thomas Richards

Aleksandra Mojsilovic

Perry S Cheng

Rachel K E Bellamy

Manish Kesarwani

Robert Leslie Yates

Kaoutar El Maghraoui

Nicholas Matthew Mitchell

Harold Leon Ossher

Stephanie Houde

David John Piorkowski

Jason Tsay

Anupama Murthi

Ganesh Venkataraman

Begum Taskazan

Jeremy Lau

Adam Welc

John Whaley

A computer implemented method of performing large-scale machine learning experiments includes expanding on one or more input datasets by systematically generating several data set drift splits. A set of experimental jobs corresponding to the generated data set drift splits are executed to generate experimental results. The experimental results are processed, consolidated, and clustered according to the generated data set drift splits.

Assistant for large-scale machine learning experiments

A technique for generating a performance prediction of a machine learning model with uncertainty intervals includes obtaining a first model configured to perform a task and a production dataset. At least one metric predicting a performance of the first model at performing the task on the production dataset is generated using a second model. The second model is a meta-model associated with the first model. At least one value predicting an uncertainty of the at least one metric predicting the performance of the first model at performing the task on the production dataset is generated using a third model. The third model is a meta-meta-model associated with the second model. An indication of the at least one metric and the at least one value is provided.

Generating performance predictions with uncertainty intervals

A method optimizes a training of a machine learning system. A conflict detection system discovers a conflict between a first training data and a second training data for a machine learning system, where the first training data and the second training data are ground truths that describe a same type of entity, and where the first training data and the second training data have different labels. In response to discovering the conflict between the first training data and the second training data for the machine learning system, an oracle adjusts the different labels of the first training data and the second training data. The machine learning system is then trained using the first training data and the second training data with the adjusted labels.

Discovering and resolving training conflicts in machine learning systems

A computer-implemented method, a computer program product, and a system for reducing labeled sample quantities required to update test sets. The computer-implemented method includes inputting a portion of unlabeled production data into a base model and generating labeled output relating to the unlabeled production data. The computer-implemented method also includes inputting the labeled output into a performance predictor. The performance predictor is a meta model of the base model that is trained with another portion of the unlabeled production data, a training set used to train the base model, and a test set portioned from the training set. The computer-implemented method further includes outputting, by the performance predictor, a performance metric relating to the labeled output produced by the trained base model. The performance metric can be any metric capable of measuring the output performance of the base model.

Label reduction in maintaining test sets

The generating of actionable recommendations for tuning model metrics of an Artificial Intelligence (AI) system includes partitioning a key performance indicator (KPI) range associated with a target system into a plurality of buckets. Log data including at least one KPI of the target system and one or more AI model metrics is partitioned and distributed across the plurality of buckets. For each bucket, an aggregate value of the one or more AI model metrics across the log data is computed and weighted according to the volume of log data in that bucket. A correlation factor between the aggregate value and a representative KPI value for each bucket is determined. A model tuning recommendation to increase ranking of the AI model metrics according to the determined correlation factor is provided to an output device and/or to the AI system for updating the one or more AI model.

Generating model insights by progressive partitioning of log data across a set of performance indicators

Methods, systems and computer program products for improving ground truth quality for modeling are provided. Aspects include receiving a plurality of data inputs, wherein each of the plurality of data inputs has an associated label. Aspects also include training a model based on the plurality of data inputs. Aspects also include generating a plurality of vector representations corresponding to the plurality of data inputs based on the model. Aspects also include clustering the plurality of vector representations into one or more clusters. Aspects also include identifying at least one anomalous data input based on the one or more clusters. The at least one anomalous data input can be a data input of the plurality of data inputs that is mislabeled, contributes to an ambiguous class structure or is an outlier. Aspects also include outputting a notification that provides an indication of the at least one anomalous data input.

Ground truth quality for machine learning models

A method for automatically generating documentation for an artificial intelligence model includes receiving, by a computing device, an artificial intelligence model. The computing device accesses a model facts policy that indicates data to be collected for artificial intelligence models. The computing device collects artificial intelligence model facts regarding the artificial intelligence model according to the model facts policy. The computing device accesses a factsheet template. The factsheet template provides a schema for an artificial intelligence model factsheet for the artificial intelligence model. The computing device populates the artificial intelligence model factsheet using the factsheet template with the artificial intelligence model facts related to the artificial intelligence model.

Generation and management of an artificial intelligence (AI) model documentation throughout its life cycle

Information flow analysis is conducted on conversational flow specifications of conversational agents. The conversational flow specification prescribes operation of a conversational agent and includes actions to be taken, inputs triggering the actions and input confirmations executed after receipt of a given input and before triggering a given action. The actions, inputs and input confirmations define conversational flow paths through the conversational flow specification. Unconfirmed conversational flow paths through the conversational flow specification are identified and reported to the author of the conversational flow specification. Each unconfirmed conversational flow path only contains inputs and actions and no input confirmations.

Information flow analysis for conversational agents

Running a global production rule on data distributed over a plurality of machines may comprise receiving a local production rule that can run on each of the plurality of machines to jointly accomplish a global computation specified by the global production rule. The local production rule may be deployed to each of the plurality of machines, each of which stores a portion of the data and runs an instance of a rules engine that can run the local production rule. The plurality of machines are enabled to communicate intermediate data produced by the instance of the rules engine running the local production rule on said each of the machines. Coordinating between the plurality of machines is enabled to synchronize one or more local computations performed locally according to the local production rule on said each machine.

Global production rules for distributed data

A programming language support for debugging heap related errors includes one or more queries for determining one or more global properties associated with use of the area by the program. The one or more queries may be executed in parallel or concurrently and dynamically utilize available number of cores.

Providing support for debugging heap related errors with heap assertions

Execution states of tasks are inferred from collection of information associated with runtime execution of a computer system. Collection of information may include infrequent samples of executing tasks, the samples which may provide inaccurate executing states. One or more tasks may be aggregated by one or more execution states for determining execution time, idle time, or system policy violations, or combinations thereof.

Automatic identification of bottlenecks using rule-based expert knowledge

Detecting defects in deployed systems, in one aspect, identify one or more monitoring agents used in a computer program. Total execution metric of the computer program and execution metric associated with the one or more monitoring agents are measured and the measure execution metric is compared with a specified overhead criteria. The execution of the one or more monitoring agents is adjusted based on the comparing step while the computer program is executing to meet the specified overhead criteria.

Detecting defects in deployed systems

A method for evaluating software performance includes steps of: receiving a plurality of versions of code; selecting starting and stopping points for timing execution of the code versions; dispatching at least two of the plurality of code versions for execution; repeatedly executing the at least two code versions; recording execution times for the at least two code versions, according to the selected starting and stopping points; collecting the execution times; and processing the collected execution times. The method further includes steps of: performing a statistical analysis of the collected execution times for determining which code version is fastest; and invoking a confidence metric periodically to determine if a difference between means is statistically meaningful.

Technique for evaluating software performance online to support online tuning

A garbage collection system that needs to meet real-time requirements breaks large objects that need to be moved into a series of arraylets. The arraylets are sized such that the amount of time required to move the arraylets is bounded by a predetermined amount. The arrays are preferably uniformly represented in the header for each array with a series of arraylet pointers that indicate the location of the arraylets. Control-flow graph splitting is used to eliminate multiple tests and allow common sub-expression elimination. Strip-mining can also be used to improve the efficiency of the process.

Method and apparatus for efficient implementation of discontiguous objects

Growing community of inventors

Ossining, NY, United States of America

Matthew R Arnold