Per-keystroke incremental lexing using a conventional batch lexer

Abstract

A system and method are disclosed that enable a batch lexer to be used to incrementally update a token stream representation of a computer program maintained in an editor as the computer program is being edited. A keystroke executive interprets editing inputs and dispatches editing events to a lexical analyzer. The lexical analyzer converts a range of the tokens likely to be affected to an equivalent old textual stream that preserves whitespace implied by but not represented within the token stream. A new text stream is generated from the old text stream by carrying out the current editing event. I.e., insertion of text is handled by the insertion of the relevant text into the old text stream (now the new stream) and deletion of a character is handled by deleting the appropriate character from the old text stream. The batch lexer is then invoked on the new text stream and as a result returns a new token stream. The fewest possible tokens from the new token stream that reflect the entire impact of the current editing event are returned to the keystroke executive along with an updated editing point within the new token stream as the suggested token stream update. The keystroke executive is free to ignore or accept the suggested update. The new token stream and the new text stream can be generated lazily, respectively one token and one character at a time.