International Business Machines Corporation

Google Technology Holdings LLC

Diptikalyan Saha

Pranay Kumar Lohia

Pankaj Dhoolia

Karthik Sankaranarayanan

Jaydeep Sen

Ashish Mittal

Aniya Aggarwal

Manish Anand Bhide

Mangala Gowri Nanda

Krishna Nandivada Venkata

Sameep Mehta

Samiulla Zakir Hussain Shaikh

Deepak Vijaykeerthy

Sandeep Hans

Manasa Jammi

Kuntal Dey

Nishtha Madaan

Rema Ananthanarayanan

Gagandeep Singh

Naveen Panwar

Vitobha Munigala

Anjan Nandy

Devika Sondhi

Eitan D Farchi

Sugata Ghosal

Geetha Adinarayan

Madhavi Katari

Anup Kumar Ghosh

Sattwati Kundu

Praduemn K Goyal

Asidhara Lahiri

Philips George John

Arpan Losalka

Swagatam Haldar

Seema Nagar

Ruchir Puri

Vibha Singhal Sinha

Kush Raj Varshney

Vinayaka D Pandit

Vijay Arya

Akshar Kaul

Ram Viswanathan

Manish Kesarwani

Priyanka Agrawal

Rajasekar Krishnamurthy

Satish Chandra

Karthikeyan Natesan Ramamurthy

Sreyash Kenkre

Parag Jain

Kahini Wadhawan

Eliran Roffe

Biswaroop Chatterjee

Ramesh Babu Tirukoti

Rachamalla Anirudh Reddy

One or more systems, devices, computer program products and/or computer-implemented methods of use provided herein relate to a process to facilitate testing a cascaded pipeline. A system can comprise a memory that stores computer executable components, and a processor that executes the computer executable components stored in the memory, wherein the computer executable components can comprise an input component, a cascaded pipeline, and an evaluation component. The input component can receive a test case associated with a label from labeled speech data represented by waveform. The evaluation component can feed the test case to the cascaded pipeline to obtain an output of the cascaded pipeline. The evaluation component can evaluate a robustness of the cascaded pipeline by comparing the output of the cascaded pipeline and the label. The cascaded pipeline can include a first model and a second model, and the first model can be different than the second model.

Testing cascaded deep learning pipelines comprising a speech-to-text model and a text intent classifier

Methods, computer program products, and systems are presented. The method computer program products, and systems can include, for instance: processing a plurality of raw datasets for generating index metadata and associating one or more index defining the index metadata to respective ones of the plurality raw datasets, wherein the respective ones of the plurality of raw datasets define respective metadata augmented datasets by the associating of the one or more index thereto; examining augmented datasets of the metadata augmented datasets in dependence on metadata of the index metadata and in dependence on user defined input data; and merging first and second ones of the augmented datasets in dependence on the examining augmented datasets, wherein the merging first and second ones of the augmented datasets in dependence on the examining augmented datasets is performed in support of preparing a production dataset.

Dataset preparation

A computer-implemented method according to one embodiment includes identifying an occurrence of accuracy drift by a trained model; identifying data associated with the accuracy drift, utilizing a drift detection model (DDM) constructed for the trained model; applying the data associated with the accuracy drift to a decision tree to determine a feature space and specific subset of the data causing the accuracy drift; analyzing a distribution of features within the feature space for the specific subset of the data causing the accuracy drift to determine specific features of the data causing the accuracy drift; and returning the specific features of the data causing the accuracy drift.

Explaining accuracy drift in production data

A processor can receive input data to a computer code and output data that the computer code produces corresponding to the input data. Based on the input data and the output data, the processor can build a decision tree that links input conditions to the output data. Using the input conditions, the processor can run a condition generator model created by fine-tuning a pre-trained programming language model, where the condition generator model predicts next condition that is likely to occur in the computer code. Based on the next condition, the processor can generate input instances and run the computer code using the generated input instances. Based on output instances output by the computer code based on the generated input instances, the processor can detect at least one flow path in the computer code, which had not been previously built in the decision tree.

Test case generation

A method, system, and computer program product for explaining predictions made by black box time series models. The method may include identifying a black box time series model. The method may also include predicting one or more time instances using the black box time series model. The method may also include selecting a predicted time instance from the predicted data. The method may also include receiving training data for the black box time series model. The method may also include generating a set of white box time series models similar to the black box time series model. The method may also include selecting a preferred white box time series model. The method may also include analyzing behavior of the preferred white box time series model. The method may also include generating an explanation illustrating why the black box time series model forecasted the predicted time instance.

Explanation for time series forecasting models

Methods, systems, and computer program products for detecting contextual bias in text are provided herein. A computer-implemented method includes identifying, by a machine learning network, a protected attribute in one or more data samples; processing the identified data samples using a first sub-network of the machine learning network, wherein the first sub-network is configured to determine a plurality of contexts of the protected attribute across the identified data samples; determining an impact of each of the plurality of contexts on a second sub-network of the machine learning network, wherein the second sub-network of the machine learning network is configured to classify a given data sample into one of a plurality of classes; and adjusting the second sub-network of the machine learning to account for the impact of at least one of the plurality of contexts on the second sub-network.

Contextual impact adjustment for machine learning models

Methods, systems, and computer program products for localization-based test generation for individual fairness testing of AI models are provided herein. A computer-implemented method includes obtaining at least one artificial intelligence model and training data related to the at least one artificial intelligence model; identifying one or more boundary regions associated with the at least one artificial intelligence model based at least in part on results of processing at least a portion of the training data using the at least one artificial model; generating, in accordance with at least one of the one or more identified boundary regions, one or more synthetic data points for inclusion with the training data; and executing one or more fairness tests on the at least one artificial intelligence model using at least a portion of the one or more generated synthetic data points and at least a portion of the training data.

Localization-based test generation for individual fairness testing of artificial intelligence models

One embodiment provides a method, comprising: receiving an input sentence for a classification by a machine-learning model, where the classification is based upon a sentiment of the input sentence; splitting the input sentence into a plurality of tokens, each of the plurality of tokens corresponding to a term within the input sentence; creating a causal subgraph from the plurality of tokens, wherein the creating is based upon a causal relationship identified between tokens of the plurality of tokens; identifying, using the causal subgraph, tokens of the plurality of tokens influencing the classification; and generating, based upon the tokens of the plurality of tokens, a causal explanation for the classification, wherein the causal explanation identifies at least one portion of the input sentence resulting in the classification.

Generation of causal explanations for text models

One embodiment provides a method, including: receiving, within a conversational agent creation framework used to create conversational agents that perform negotiations on behalf of users, (i) a selection of one of a plurality of conversational agent roles (ii) negotiation constraints for the selected role, and (iii) an object for the negotiation; creating a conversational agent (i) having the selected role and (ii) programmed with the negotiation constraints; identifying at least one other conversational agent (i) having an opposing role with respect to the object and (ii) being programmed with opposing role negotiation constraints; and conducting a conversational session between the conversational agent and the at least one other conversational agent, wherein the conversational session comprises the negotiation for the object and wherein the conversational agent and the at least one other conversational agent perform the negotiation in view of the negotiation constraints and the opposing role negotiation constraints.

Multi-agent conversational agent framework

An example system includes a processor to receive a set of features, a set of relations between the features, and a set of target features. Each of the target features is associated with a number of the relations. The processor can generate a hypergraph based on the features and the relations. The processor also can select a subset of features based on a transitive closure of the hypergraph for each of the target features. The processor can transmit the selected subset of features.

Feature selection using hypergraphs

Methods, systems, and computer program products for automatically testing AI models in connection with enterprise-related properties are provided herein. A computer-implemented method includes obtaining an artificial intelligence model and information pertaining to one or more existing enterprise-related properties associated with the artificial intelligence model; generating test data for the artificial intelligence model by processing at least a portion of the information pertaining to one or more existing enterprise-related properties; executing at least a portion of the generated test data against the artificial intelligence model; generating, based at least in part on results of the executing of the at least a portion of the generated test data against the artificial intelligence model, one or more enterprise-related property recommendations for the artificial intelligence model; and performing one or more automated actions based at least in part on the one or more enterprise-related property recommendations.

Automatically testing artificial intelligence models in connection with enterprise-related properties

An embodiment for identifying and sorting duplicate datasets within a large pool of heterogeneous datasets may include received a plurality of heterogeneous datasets. The embodiment may automatically compare schema information and metadata within each of the received plurality of heterogeneous datasets to generate name-based similarity scores for each dataset. The embodiment may also automatically compare data distribution information within each of the received plurality of heterogeneous datasets to generate a plurality of data distribution similarity scores for each heterogeneous dataset. The embodiment may further include automatically calculating an overall distance metric using the name-based similarity scores and plurality of data distribution similarity scores. The embodiment may also include based on the calculate overall distance metric, automatically generating distance graphs that identifying clusters of similar datasets and illustrate inferred lineage for the clusters of similar datasets.

Scalable identification of duplicate datasets in heterogeneous datasets

A system, computer program product, and method are presented for providing replacement data for data in a time series data stream that has issues indicative of errors, where the data issues and the replacement data are related to one or more KPIs. The method includes determining one or more predicted replacement values for potentially erroneous data instances in the time series data stream. The method further includes resolving the potentially erroneous data instances with one predicted replacement value of the one or more predicted replacement values in the time series data stream.

Data quality-based computations for KPIs derived from time-series data

Methods, systems, and computer program products for path-coverage directed black box application programming interface (API) testing are provided herein. A computer-implemented method includes determining constraints based on inputs and corresponding outputs of an API in a production environment; generating initial test inputs based at least in part on the constraints; creating a program dependency graph based on trace sequences and request-response data obtained in response to providing the initial test inputs to an endpoint of the API; enhancing the program dependency graph by generating additional test inputs directed to one or more paths of the dependency graph; identifying, based on the enhanced program dependency graph, at least a portion of the API that is not covered by an existing test suite; and using the enhanced program dependency graph to generate new test cases for the test suite based on the identifying.

Path-coverage directed black box API testing

Methods, systems, and computer program products for automatically modifying responses from generative models using artificial intelligence techniques are provided herein. A computer-implemented method includes obtaining data pertaining to at least one conversation involving at least one automated conversation exchange software program and at least one user; identifying, among words proposed by the at least one automated conversation exchange software program in connection with the at least one conversation, words qualifying as belonging to one or more predetermined categories by processing the obtained data using artificial intelligence techniques; determining, by processing the identified words and at least one word-based data source, one or more alternate words; modifying at least a portion of the proposed words by replacing at least a portion of the identified words with at least a portion of the one or more alternate words; and performing at least one automated action based on the modifying.

Automatically modifying responses from generative models using artificial intelligence techniques

Growing community of inventors

Bengaluru, India

Diptikalyan Saha