Microsoft Technology Licensing, LLC

Michael Shilman

Sashi Raghupathy

Zile Wei

Charlton E Lui

Paul A Viola

F David Jones

David Bargeron

Patrice Y Simard

Herry Sutanto

Jian Wang

Desney S Tan

Bodin Dresevic

Yu Zou

Richard Jennings Duncan

Zoltan C Szilagyi

Timothy H Kannapel

Jamie N Wakeam

Tomer Moscovich

Kenneth Paul Hinckley

John Carlton Platt

Patrick Markus Baudisch

Raman Kumar Sarin

Kumar Hemachandra Chellapilla

Alex Jacob Simmons

Peter Loren Engrav

Steve Dodge

Maneesh Agrawala

Kevin Phillip Paulson

Edward Bryan Cutrell

Mukund Narasimhan

Jack W Stokes

Jerome J Turner

Todd M Landstad

Ming Ye

Thomas R Wick

Kentaro Urata

Shengdong Zhao

Joseph L Kravis

Chengyang Li

David M Bargeron

A two-dimensional representation of a document is leveraged to extract a hierarchical structure that facilitates recognition of the document. The visual structure is grammatically parsed utilizing two-dimensional adaptations of statistical parsing algorithms. This allows recognition of layout structures (e.g., columns, authors, titles, footnotes, etc.) and the like such that structural components of the document can be accurately interpreted. Additional techniques can also be employed to facilitate document layout recognition. For example, grammatical parsing techniques that utilize machine learning, parse scoring based on image representations, boosting techniques, and/or 'fast features' and the like can be employed to facilitate in document recognition.

Grammatical parsing of document visual structures

A digital ink annotation process and system for processing digital documents and digital ink annotations therein. An annotation's position is maintained within a document such that the original intent and meaning of the annotation is preserved. This is true even if the document is edited, resized, displayed on a different device or otherwise modified. The process includes automatic and manual grouping of digital ink strokes within a document to define digital ink annotations, classifying the annotations according to annotation type, and anchoring the annotations to appropriate regions or positions in a document. The process further includes reflowing the annotations in a new document layout such that the annotations conform and adapt to the new layout while preserving the original intents and meanings of the annotations. The system includes a classification module, an anchoring module, a reflow module and a clean-up module to implement the digital ink annotation process.

Recognizing, anchoring and reflowing digital ink annotations

A malicious behavior detection/prevention system, such as an intrusion detection system, is provided that uses active learning to classify entries into multiple classes. A single entry can correspond to either the occurrence of one or more events or the non-occurrence of one or more events. During a training phase, entries are automatically classified into one of multiple classes. After classifying the entry, a generated model for the determined class is utilized to determine how well an entry corresponds to the model. Ambiguous classifications along with entries that do not fit the model well for the determined class are selected for labeling by a human analyst. The selected entries are presented to a human analyst for labeling. These labels are used to further train the classifier and the models. During an evaluation phase, entries are automatically classified using the trained classifier and a policy associated with determined class is applied.

Method of classifying and active learning that ranks entries based on multiple scores, presents entries to human analysts, and detects and/or prevents malicious behavior

Flexible and efficient systems and methods for organizing, analyzing, and processing digital ink incrementally analyze input data (e.g., representing ink strokes) as the user continues to add to, edit, or modify the data. In this manner, processing is performed promptly as the ink is entered, and the processing system and method can effectively keep up with the user. This prevents long processing delays, because the systems and methods need not first process a large volume of ink data present after the user has entered has completely filled a page with ink.

Incremental system for real time digital ink analysis

The present invention leverages spatial relationships to provide a systematic means to recognize text and/or graphics. This allows augmentation of a sketched shape with its symbolic meaning, enabling numerous features including smart editing, beautification, and interactive simulation of visual languages. The spatial recognition method obtains a search-based optimization over a large space of possible groupings from simultaneously grouped and recognized sketched shapes. The optimization utilizes a classifier that assigns a class label to a collection of strokes. The overall grouping optimization assumes the properties of the classifier so that if the classifier is scale and rotation invariant the optimization will be as well. Instances of the present invention employ a variant of AdaBoost to facilitate in recognizing/classifying symbols. Instances of the present invention employ dynamic programming and/or A-star search to perform optimization. The present invention applies to both hand-sketched shapes and printed handwritten text, and even heterogeneous mixtures of the two.

Spatial recognition and grouping of text and graphics

A handwriting recognition system interprets handwritten text and produces a typed interpretation of that text. When the initial interpretation of the handwritten text is inaccurate, the handwriting recognition system alters the initial recognition by reinterpreting the handwritten text in view of a correction made by a user and constraints (e.g., derived by assumptions in user behavior). The handwriting recognition system intelligently reinterprets and renews its text recognition each time the user implements a correction. In effect, a single correction can trigger multiple adjustments to the text recognition. Therefore, with the use of a reinterpretation algorithm, the handwriting recognition system helps the user obtain the desired result in fewer correction steps.

Constraint-based correction of handwriting recognition errors

A system and method that facilitates and effectuates in situ search for active note taking. The system and method includes receiving gestures from a stylus and a tablet associated with the system. Upon recognizing the gesture as belonging to a set of known and recognized gestures, the system creates an embeddable object, initiates a search with terms indicated by the gesture, associates the search results with the created object and inserts the object in close proximity with the terms that instigated the search.

In situ search for active note taking

Techniques for efficiently and accurately organizing freeform handwriting into lines. A global cost function is employed to find the simplest partitioning of electronic ink strokes into line groups that also maximize the 'goodness' of the resulting lines and the consistency of their configuration. The 'goodness' of a line may be based upon its linear regression error and the horizontal and vertical compactness of the strokes making up the line. The line consistency configuration for a grouping of strokes is measured by the angle difference between neighboring groups. The global cost function also takes into account the complexity of the stroke partitioning, measured by the number of lines into which the strokes are grouped. An initial grouping of strokes is made, and the cost for this initial grouping is determined. Alternate groupings of the initial stroke grouping are then generated. The global cost of each of these alternate stroke groupings is then calculated, and the stroke grouping that produces the largest global cost decrease from the global cost of the original grouping is selected. The alternate grouping creation, cost determination and evaluation, and grouping selection process then is repeated until the global cost for new grouping alternates no longer decreases.

Grouping lines in freeform handwritten text

Image recognition is utilized to facilitate in scoring parse trees for two-dimensional recognition tasks. Trees and subtrees are rendered as images and then utilized to determine parsing scores. Other instances of the subject invention can incorporate additional features such as stroke curvature and/or nearby white space as rendered images as well. Geometric constraints can also be employed to increase performance of a parsing process, substantially improving parsing speed, some even resolvable in polynomial time. Additional performance enhancements can be achieved in yet other instances of the subject invention by employing constellations of integral images and/or integral images of document features.

Application of grammatical parsing to visual recognition tasks

The present invention leverages classification type detectors and/or context information to provide a systematic means to recognize and anchor annotation strokes, providing reflowable digital annotations. This allows annotations in digital documents to be archived, shared, searched, and easily manipulated. In one instance of the present invention, an annotation recognition method obtains an input of strokes that are grouped, classified, and anchored to underlying text and/or points in a document. Additional instances of the present invention utilize linguistic content, domain specific information, anchor context, and document context to facilitate in correctly recognizing an annotation.

Freeform digital ink annotation recognition

Dynamic inference is leveraged to provide online sequence data labeling. This provides real-time alternatives to current methods of inference for sequence data. Instances estimate an amount of uncertainty in a prediction of labels of sequence data and then dynamically predict a label when an uncertainty in the prediction is deemed acceptable. The techniques utilized to determine when the label can be generated are tunable and can be personalized for a given user and/or a system. Employed decoding techniques can be dynamically adjusted to tradeoff system resources for accuracy. This allows for fine tuning of a system based on available system resources. Instances also allow for online inference because the inference does not require knowledge of a complete set of sequence data.

Continuous inference for sequence data

The present invention relates to systems and methods that facilitate annotating digital documents (e.g., digital inking) with devices such as Tablet PCs, PDAs, cell phones, and the like. The systems and methods provide for multi-scale navigation during document annotating via a space-scale framework that fluidly generates and moves a zoom region relative to a document and writing utensil. A user can employ this zoom region to annotate various portions of the document at a size comfortable to the user and suitably scaled to the device display. The space-scale framework enables dynamic navigation, wherein the zoom region location, size, and shape, for example, can automatically adjust as the user annotates. When the user finishes annotating the document, the annotations scale back with the zoom region to original page size. These novel features provide advantages over conventional techniques that do not contemplate multi-scale navigation during document annotating.

Systems and methods that utilize a dynamic digital zooming interface in connection with digital inking

Systems, methods, and computer-readable media for making rich, flexible, and more natural electronic ink annotations in an electronic document include creating a first context node associated with a first portion of a base portion of an electronic document; creating a second context node associated with an annotation to the base portion; and linking the second context node with the first context node.

Electronic ink processing

Electronic ink layout analysis systems and methods provide flexibility and efficiency in organizing, analyzing, and processing digital ink. These layout analysis systems and methods allow users substantial freedom in entering electronic ink into a pen-based computer system. Using these systems and methods, a user's input digital ink is not constrained by requirements that a user write in a specific screen orientation, that a user write in one specific orientation on all portions of a page, or that a user write using a specific minimum or maximum sized stroke. Rather, the systems and methods freely allow the user to write anywhere on a given page, in any orientation or size, while still enabling effective and efficient handwriting recognition and other processing of the input digital ink.

Handwriting layout analysis of freeform digital ink input

A method of analyzing electronic ink, in which document data for a document containing electronic ink content is received from a software application running on a first processing thread. The first processing thread is employed to provide the document data to an electronic ink analysis process for analyzing on a second processing thread. Control of the first processing thread is then returned to the software application. After the results of the analysis are received, the results are reconciled with the current document data for the document.

Growing community of inventors

Seattle, WA, United States of America

Michael Shilman