Microsoft Technology Licensing, LLC

Crowdflower, Inc.

Figure Eight Technologies, Inc.

Lukas A Biewald

Chad P Walters

Christopher R Van Pelt

Giovanni Lorenzo Thione

Barney Pell

Brendan O'Connor

Nitay Joffe

Andrew Alan James

Scott A Prevost

David Oleson

Greg Laughlin

Alexander Sorokin

John Le

Vaughn Hester

Systems and methods for using virtual currency to compensate workers in a crowd sourced task are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, of determining target accuracy in completion of the task by the online workers, computing a number of results to be collected for a unit job of the task based on the target accuracy; and/or setting a wage amount that is offered to a worker for completing a unit job of the task based on the number of results. In response to determining that the worker has completed the unit job, the wage is provided to the online worker in a form of virtual currency.

Using virtual currency to compensate workers in a crowdsourced task

Systems and methods of crowd sourcing by distributing a task over a network to multiple workers for completion while providing quality control are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, of identifying an attribute of the task, the task including multiple unit jobs, determining target accuracy in completion of the task by the multiple workers, computing a number of results to be collected for a unit job of the task based on the target accuracy, setting a wage amount that is offered to a worker for completing a unit job of the task based on the attribute and the target accuracy, and/or collecting and aggregating statistics of the workers completing the select unit jobs for quality control.

Distributing a task to multiple workers over a network for completion while providing quality control

Tools and techniques are described herein for checkpointing iterators during search. These tools may provide methods that include instantiating iterators in response to a search request. The iterators include fixed state information that remains constant over a life of the iterator, and further include dynamic state information that is updated over the life of the iterator. The iterators traverse through postings lists in connection with performing the search request. As the iterators traverse the posting lists, the iterators may update their dynamic state information. The iterators may then evaluate whether to create checkpoints, with the checkpoints including representations of the dynamic state information.

Checkpointing iterators during search

Systems and methods for evaluating a worker in performing crowd sourced tasks and providing in-task training are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, for selecting a job distributed through a job distribution platform for workers to work on, for use to generate a test task, the job being associated with a known correct result, associating a manipulated result, known to be an incorrect result for the job, with the job to generate the test task, and/or presenting the job with the manipulated result as the test task to a worker for evaluation of work quality of the worker. The job distribution platform crowd sources tasks online to workers to work on via their respective computing devices.

Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks

A role tree having nodes corresponding to semantic roles in a hierarchy is defined. A posting list is generated for each association of a term and a semantic role in the hierarchy. The posting lists are stored contiguously on a physical storage medium such that a subtree of the hierarchy of semantic roles can be loaded from the storage medium as a single contiguous block. The posting lists for a subtree of the hierarchy are retrieved by obtaining data identifying the beginning location on the physical storage medium of the posting lists for the term at the top of a desired subtree of the hierarchy and data identifying the length of the posting lists of the desired subtree of the hierarchy. A single contiguous block that includes the posting lists for the desired subtree of the hierarchy is then retrieved from the beginning location through the specified length.

Efficient storage and retrieval of posting lists

Computer-readable media, computerized methods, and computer systems for conducting semantic processes to present search results that include highlighted regions which are relevant to a conceptual meaning of a query are provided. Initially, content of document(s) is accessed and semantic representations are derived by distilling linguistic representations from the content. These semantic representations may be stored at a semantic index. Also, a proposition is derived from the query by parsing search terms of the query, and distilling the proposition from the search terms. Typically, the proposition is a logical representation of the conceptual meaning of the query. The proposition is compared against the semantic representations at the semantic index to identify a matching set. Regions of the content within the document, from which the matching set of semantic representations are derived, are targeted. Accordingly, highlighting may be applied to the targeted regions when presenting or displaying the search results.

Growing community of inventors

San Francisco, CA, United States of America

Lukas A Biewald