Google Inc.

Microsoft Technology Licensing, LLC

Semantic Machines, Inc.

Percy Shuo Liang

Jakob David Uszkoreit

Daniel Martin Bikel

David Leo Wright Hall

Joshua James Clausman

Daniel Louis Klein

Jesse Daniel Eskes Rusak

Pravir Kumar Gupta

Ciprian Ioan Chelba

Laurence Steven Gillick

Jordan Rian Cohen

Omer Bar-or

Adam David Pauls

Linda Kathleen Arsenault

A method for training a dialogue learning model includes presenting, via a user interface of a computing device, an utterance and a list of actions based on the utterance. A selection of an action from the list of actions is received via the user interface. A designated span of the utterance is received via the user interface. The selected action and the designated span of the utterance is provided to a computing system for training the dialogue learning model.

Expandable dialogue system

An automated natural dialogue system provides a combination of structure and flexibility to allow for ease of annotation of dialogues as well as learning and expanding the capabilities of the dialogue system based on natural language interactions.

Robust expandable dialogue system

A system that allows non-engineers administrators, without programming, machine language, or artificial intelligence system knowledge, to expand the capabilities of a dialogue system. The dialogue system may have a knowledge system, user interface, and learning model. A user interface allows non-engineers to utilize the knowledge system, defined by a small set of primitives and a simple language, to annotate a user utterance. The annotation may include selecting actions to take based on the utterance and subsequent actions and configuring associations. A dialogue state is continuously updated and provided to the user as the actions and associations take place. Rules are generated based on the actions, associations and dialogue state that allows for computing a wide range of results.

A method for generating training data for training a natural language processing system comprises loading, into a computer memory, a computer-readable transcript representing an ordered sequence of one or more dialogue events. The method further comprises acquiring a computer-readable command describing an exemplary ordered subsequence of one or more dialogue events from the computer-readable transcript. The method further comprises re-parametrizing the computer-readable command with an alternative semantic parameter. The method further comprises generating an alternative ordered subsequence of one or more dialogue events based on the re-parametrized computer-readable command. The method further comprises outputting, to a data store, an alternative computer-readable transcript including the alternative ordered subsequence of one or more dialogue events, the alternative computer-readable transcript having a predetermined format usable to train the computerized assistant.

Training natural language system with generated dialogues

Flexible and expandable dialogue system

A computing device includes a display configured to present a graphical user interface. The graphical user interface includes a transcript portion configured to display an unannotated transcript representing an ordered sequence of one or more dialogue events involving a client and a computerized assistant, at least one of the dialogue events taking the form of an example client utterance, and an annotation portion configured to display a hierarchical menu including a plurality of candidate utterance annotations. An utterance annotation machine is configured to receive one or more computer inputs selecting, for each of one or more response parameters in the example client utterance, utterance annotations from the hierarchical menu that collectively define a machine-readable interpretation of the example client utterance. An annotated utterance having a predetermined format usable to train the computerized assistant is output to a data store based on the example client utterance.

Utterance annotation user interface

A data collection system is based on a general set of dialogue acts which are derived from a database schema. Crowd workers perform two types of tasks: (i) identification of sensical dialogue paths and (ii) performing context-dependent paraphrasing of these dialogue paths into real dialogues. The end output of the system is a set of training examples of real dialogues which have been annotated with their logical forms. This data can be used to train all three components of the dialogue system: (i) the semantic parser for understanding context-dependent utterances, (ii) the dialogue policy for generating new dialogue acts given the current state, and (iii) the generation system for both deciding what to say and how to render it in natural language.

Data collection for a new conversational dialogue system

A language processing system identifies first command input sentences that do not successfully parse by any parsing rule in a set of parsing rules. Each of the parsing rules is associated with an action, and a user device performs the action associated with a parsing rule in response to an input sentence being successfully parsed by the parsing rule. For each of these identified first sentences, the system determines whether the first input sentence has an underserving signal that is indicative of one or more actions being underserved. If the first sentence has the underserving signal, then the first sentence is selected as a candidate input sentence. Each candidate input sentence is provided to an action analysis processes that determines whether a candidate input sentence is to be associated with one action, and upon a positive determination generates a parsing rule for the candidate input sentence.

Identifying underserved command inputs

Systems, methods and apparatus for learning parsing rules and argument identification from crowdsourcing of proposed command inputs. Crowdsourcing techniques to generate rules for parsing input sentences. A parse is used to determine whether the input sentence invokes a specific action, and if so, what arguments are to be passed to the invocation of the action.

Learning parsing rules and argument identification from crowdsourcing of proposed command inputs

Systems, methods and apparatus for learning parsing rules and argument identification from crowdsourcing of proposed command inputs are disclosed. Crowdsourcing techniques are used to generate rules for parsing input sentences. A parse is used to determine whether the input sentence invokes a specific action, and if so, what arguments are to be passed to the invocation of the action.

A language processing system identifies sequential command inputs in user session data stored in logs. Each sequence command input is a first command input followed by a second command input. The system determines user actions in response to each command input. For the second command input, an action was taken at the user device in response to the command input, and there is no parsing rule associated with the action that parses to the first command input. If there is a sufficient co-occurrence of the first and second command inputs and the resulting action in the logs, then a parsing rule for the action may be augmented with a rule for the first command input.

Parsing rule augmentation based on query sequence and action co-occurrence

A language processing system uses a lattice parser that semantically parses a command input represented by a lattice. The parser receives a hypotheses space of outputs as encoded in a lattice. Annotations of the input are projected back into the lattice and then lattice parsing is performed to rectify with the annotations. Parsing rules are applied to path fragments in the lattice. The rules that successfully parse from the start node to the end node of the lattice are used to determine whether the command input sentence invokes a specific action, and if so, what arguments are to be passed to the invocation of the action.

Growing community of inventors

Palo Alto, CA, United States of America

Percy Shuo Liang