Microsoft Technology Licensing, LLC

International Business Machines Corporation

Saleema Amin Amershi

Patrice Y Simard

David G Grangier

Bongshin Lee

Aparna Lakshmiratan

Denis X Charles

Adam Fourney

David Max Chickering

Victor Chukwuma Dibia

Steven Mark Drucker

Ashish Kapoor

Meredith June Morris

Jaime Brooks Teevan

Carlos Garcia Jurado Suarez

Leon Bottou

Forough Poursabzi Sangdeh

David Maxwell Chickering

Jalal U Mahmud

Danyel Aharon Fisher

Ratul Mahajan

Tessa Ann Lau

Jason D Williams

Jeffrey William Nichols

James Wiliamson Mickens

Kristina Nikolova Toutanova

Daniel John Liebling

Semiha Ece Kamar Eden

Robert Warren Gruen

Pallavi Choudhury

Shamsi Tamara Iqbal

Jin A Suh

Ann M Paradiso

Donghao Ren

Anthony Lee Carbary

Gagan Bansal

Rich Caruana

Blaine S Christian

Darren Francis Gehring

Jina Suh

Todd Kulesza

Michael J Brooks

Patrice Yvon Rene Simard

James W Mickens

Systems and methods for directing behavior of a generative artificial intelligence (AI) system are provided. In particular, a computing device may obtain an input prompt associated with a requested task for one or more generative artificial intelligence (AI) systems, obtain one or more attributes based on the input prompt, modify the input prompt based on the one or more embedded attributes, and provide the modified input prompt to the one or more generative AI systems.

Embedded attributes for modifying behaviors of generative AI systems

Solutions for evaluating source code generators use offline and online evaluation stages. Offline evaluation includes separating each of a plurality of input passages of software code into a plurality of constituent blocks. Each code generator (of a plurality of code generators) generates an equivalent block corresponding to each constituent block. A coding score is determined for each equivalent block (for each code generator), and the coding scores are aggregated across the equivalent blocks to provide an aggregate score for each code generator. A ranking of the aggregate scores is used to down-select to a fewer number of code generators for online evaluation. For this stage, the code generators output passages of software code, and user acceptance of the code generators' outputs may be used for further ranking and down-selection. Some examples weight the coding score according to a code utility estimate of the constituent blocks for which equivalent blocks are generated.

Coding activity task (CAT) evaluation for source code generators

A collection of data that is extremely large can be difficult to search and/or analyze. Relevance may be dramatically improved by automatically classifying queries and web pages in useful categories, and using these classification scores as relevance features. A thorough approach may require building a large number of classifiers, corresponding to the various types of information, activities, and products. Creation of classifiers and schematizers is provided on large data sets. Exercising the classifiers and schematizers on hundreds of millions of items may expose value that is inherent to the data by adding usable meta-data. Some aspects include active labeling exploration, automatic regularization and cold start, scaling with the number of items and the number of classifiers, active featuring, and segmentation and schematization.

Interactive feature selection for training a machine learning system and displaying discrepancies within the context of the document

Edits on a content item, such as a document, are divided into microtasks. The microtasks associated with a document can be automatically identified based on a workflow or can be identified by a user associated with the content item or an administrator. At a later time, the user can complete the microtasks for a content item using an application associated with their smart phone or tablet. The application may present the microtasks in a game-like environment where the user can compete with other users based on metrics such as number of microtasks completed in a day or fastest completion time. In addition, the user can earn rewards such as badges, coupons, or credits by completing microtasks. In this way, users can use time that would have been wasted playing games to complete their content items, while still experiencing some of the fun and competition associated with the games.

Providing rewards and metrics for completion of microtasks

Methods, computer systems, computer-storage media, and graphical user interfaces are provided for improving performance of a multi-class classifier. An interactive graphical user interface includes an item representation display area that displays a plurality of item representations corresponding to a plurality of items processed by a multi-class classifier. The classifier's performance can be visualized using bidirectional bar graphs displaying true positives, false positives, and false negatives for each class.

Interactive performance visualization of multi-class classifier

Interactive concept editing in computer-human interactive learning

A system, method, and media are provided for generating a structured labeling graphical user interface. The user interface receives user input that associates multimedia content with categories. The user input may include user-supplied tags that further define the category for the multimedia content. The user-supplied tags are rendered proximate to the categories. In turn, a database logs user events to store, among other things, the categories, the user-supplied tags, time associated with completing the user-supplied tags, and time for associating multimedia content with the categories or tags.

Structured labeling to facilitate concept evolution in machine learning

Disclosed herein are technologies directed to a feature ideator. The feature ideator can initiate a classifier that analyzes a training set of data in a classification process. The feature ideator can generate one or more suggested features relating to errors generated during the classification process. The feature ideator can generate an output to cause the errors to be rendered in a format that provides for an interaction with a user. A user can review the summary of the errors or the individual errors and select one or more features to increase the accuracy of the classifier.

Error-driven feature ideation in machine learning

Methods, computer systems, computer-storage media, and graphical user interfaces are provided for visualizing a performance of a machine-learned model. An interactive graphical user interface includes an item representation display area that displays a plurality of item representations corresponding to a plurality of items processed by the machine-learned model. The plurality of item representations are arranged according to scores assigned to the plurality of items by the machine-learned model. Further, each of the plurality of item representations is visually configured to represent a label assigned to a corresponding item.

Interactive visualization of machine-learning performance

Embodiments of the invention relate to generating automated web task procedures from an analysis of web history logs. One aspect of the invention concerns a method that comprises identifying sequences of related web actions from a web log, grouping each set of similar web actions into an action class, and mapping the sequences of related web actions into sequences of action classes. The method further clusters each group of similar sequences of action classes into a cluster, wherein relationships among the action classes in the cluster are represented by a state machine, and generates automated web task procedures from the state machine.

Automated web task procedures based on an analysis of actions in web browsing history logs

Interactive segment extraction in computer-human interactive learning

Active featuring in computer-human interactive learning

The described implementations relate to data classification. One implementation includes identifying one or more likely classifications for an incoming data item using an algorithm. The implementation can also include providing the one or more identified classifications to a user. A selection of an individual identified classification for the incoming data item can be received from the user. The algorithm can be refined to reflect the selection by the user.

Classification of stream-based data using machine learning

Systems and methods are provided to perform collaborative retrieval, communication, and navigation of electronic content in a co-located environment. In an illustrative implementation, a collaborative content environment comprises a collaborative content interface engine, and an instruction set comprising at least one instruction providing instructions to the collaborative content interface engine to process data representative of inputs from two or more cooperating interface devices to allow for the retrieval, communication, search, and navigation of electronic content. In the illustrative implementation, the collaborative content interface engine can present retrieved, communicated, searched, and/or navigated data according to a selected display paradigm. The display paradigm can include one or more display portions of a display pane comprising data responsive to the inputs received from the two or more cooperating interface devices.

System and interface for co-located collaborative web search

Growing community of inventors

Seattle, WA, United States of America

Saleema Amin Amershi