Token-based computer program editor with program comment management

Abstract

A user-friendly text editor for structurally represented computer programs is disclosed. The present editor combines advantages of text editors and structure editors by transforming, as the user types, the text stream entered by the user into a token stream, where the tokens of the token stream constitute the words of the program being entered. Each of the tokens is classified by the editor as one of a group of extended lexemes defined by the language in which the program being edited is written. These extended lexemes are defined similar to lexemes that might be used in a batch lexer, but are more numerous as the present editor must account for the incomplete and ill-formed lexemes that arise as the user types the program. Before performing lexical analysis, the present editor separates program statements from program comments. The editor then stores the text of the comments along with comment positioning information that allows the original position of the comments as entered by the user to be recreated for printing or display purposes. Based on the token stream and the separately represented comments, the editor prettyprints the program and comments as the user types. The editor also allows the user to edit the program from the prettyprinted display as if the program were internally represented as text. The present editor also allows the user to periodically request a syntactical analysis of the program, wherein the structure of the program is represented in a syntax tree that references the token stream.