System and method for inter-token whitespace representation and textual

Abstract

A user-friendly editor for structurally represented computer programs is disclosed. The present editor combines advantages of text editors and structure editors by transforming, as the user types, the text stream entered by the user into a token stream, where the tokens of the token stream constitute the words of the program being entered. Each of the tokens is classified by the editor as one of group of extended lexemes defined by the language in which the program being edited is written. These extended lexemes are defined similarly to lexemes that might be used in a batch lexer, but are more numerous as the present editor must account for the incomplete and ill-formed lexemes that arise as the user types the program. Based on information in the token stream, the editor prettyprints the program as the user types. This prettyprinting step involves typesetting each of the tokens in a visually distinct manner and displaying a varying amount of visual whitespace between the tokens, each of these operations being based on the lexical types of the tokens. The editor also allows the user to edit the program from the prettyprinted display as if the program were internally represented as text via a cursor whose position and appearance depends on the lexical types of tokens adjacent to the cursor. The present editor also allows the user to periodically request a syntactical analysis of the program, wherein the structure of the program is represented in a syntax tree that references the token stream.