COOLEDIT - TEXT EDITOR AND INTEGRATED DEVELOPMENT ENVIRONMENT ------------------------------------------------------------- INTRODUCTION ------------ In 1996 work began on an internal editor for the Midnight Commander to mimic and extend the editor of the Norton Commander DOS file manager. At the same time, the author's thesis project led to writing a widget library called Coolwidgets for X (the stereo-0.2 package available on the sunsite). This evolved into a text editor that had a terminal interface via the Midnight Commander and an X interface via Coolwidgets. Today, Cooledit has matured into a refined and sophisticated programming environment. Important to Cooledit's development was that it was written mostly using itself, and has hence seen more work in its own creation than by any other single need. This has important implications (besides the obvious existential ones) and will be discussed with reference to general GUI design. Cooledit comes with several other utilities written using the Coolwidgets library. It is written entirely in ANSI C. COOLEDIT AND COOLWIDGET ARCHITECTURE ------------------------------------ It should be stated at the outset that Coolwidgets is poorly designed. Although extensive and repeated overhauls have been made to the source, and although most of the individual functions are cleanly written, Coolwidgets is by and large a `hack'. It lacks proper modularity and extensibility. Several important lessons can be learned from this. Coolwidgets was an attempt to create a widget library that worked in parallel with XLib, the idea being that the programmer could have the low level control of X, combined with the high level shortcuts of a ready library. When this became messy, it leaned toward a slightly object oriented design. It is successful in that it is the minimalist widget library for its requirements - that is, it is probably the smallest piece of code possible for what it has to do. How Coolwidgets actually works is best described with a `hello world' example: /* hello.c - simple example usage of the coolwidget X API */ #include int main (int argc, char **argv) { Window win; CInitData cooledit_startup; CEvent cwevent; int y; /* initialize the library */ memset (&cooledit_startup, 0, sizeof (cooledit_startup)); cooledit_startup.name = argv[0]; /* won't bother with other init's like geom and display */ cooledit_startup.font = "-*-helvetica-bold-r-*--14-*-*-*-p-*-iso8859-1"; CInitialise (&cooledit_startup); /* create main window */ win = CDrawMainWindow ("hello", "Hello"); CGetHintPos (0, &y); /* y position where to start drawing */ CDrawText ("hellotext", win, 0, y, " Hello World "); /* the lable "hellotext" may be used to identify the widget later */ CCentre ("hellotext"); /* we want the text centred from left to right */ CGetHintPos (0, &y); /* get the next y position below the last widget drawn */ CDrawButton ("done", win, 0, y, AUTO_SIZE, " Done "); /* draw a button... */ CCentre ("done"); /* ...centred */ CSetSizeHintPos ("hello"); /* set the window size to just fit the widgets */ CFocus (CIdent ("done")); /* show the window */ CMapDialog ("hello"); /* Run the application. */ do { CNextEvent (0, &cwevent); if (cwevent.type == QuitApplication) /* pressed WM's "close" button */ break; } while (strcmp (cwevent.ident, "done")); /* close connection to the X display */ CShutdown (); return 0; } Coolwidgets widgets are stored internally and referenced by an identifier string. The string must be unique within the scope of the application. This identifier approach was appealing because the programmer need not declare every widget created, and can easily globally reference any widget - it has no other real advantages though. The workhorse of the application is CNextEvent() - a wrapper around XNextEvent that does things like handle Expose events, key-press translation and widget call-backs. Using CNextEvent(), Coolwidgets can be used in a straight C fashion like above, or in callback (ala object-orientated) fashion with the CAddCallback() function. In this way it is quite versatile. A widget is defined by a 512 byte structure. There is only one structure for all the different types of widgets - fields were just added as needed, hence most of the fields are not used (the memory waist incurred by this is not significant). All the widgets are allocated into an array within the library. The editor itself is just an additional widget of the library, in the same way as it is under the Midnight Commander. The Cooledit application contains the Python interpretor, shell and gdb interfaces, and manages the multiple window interface. The library has some peculiar features: - Additional events may be received by CNextEvent besides the usual X events. QuitApplication above is an example (wm close). AlarmEvent and EditorCommand are others. These provide additional functionality. - Exposes are amalgamated into larger exposes for efficiency. Application need not do this themselves and can rely on received expose areas being optimal. - Key presses are translated into editor commands before reaching the application. These are high level commands like Cut and Paste. Hence any application written with Coolwidgets will have the same key-bindings. Key-bindings are consistent through all widgets. Cooledit also implements a way to redefine keys globally, by applying a key translator callback at a low level. - Dialogs and menus dynamically assign hotkeys (underlined letters) from heuristics. This means that international languages will have hotkeys without translators having to explicitly work out a hotkey arrangement. Alt- combinations are an intrinsic part of Coolwidgets. - All entry widgets have a history. At the moment the application has to take the trouble to load and save this history, but if it does, then the user has the benefit of persistent storage of every entry ever made. DESIGNING AN INTERFACE ---------------------- It is noteworthy that the author had extensive use of a myriad of text editors before beginning this project. Each of them had a set of nifty features along with just as many irritating ones. Cooledit attempts to be the combination of the best behavior from all these editors. Although logic plays an important role when creating an interface, it cannot completely anticipate the psychology of users' reactions. A user will tend to avoid using some feature that they find irritating (or because they are used to doing it differently in another program). The user will try a combination of other features (or even another program) to achieve the same result. They may get used to doing things a different way, and will expect their own approach to be ergonomic, even if it conflicts with the intention of the developer. A users use of an application will not necessarily propagate to the minimal set of key strokes just because an exhaustive combination of features are present. How users will end up using an application is highly unpredictable. The solution is two fold: 1. The developers should have extensive use of many other similar applications. This does not just mean looking at what features an application has and how they are invoked, but also using those features extensively until the operations become spontaneous. Only at this point is the application fully evaluated. 2. Create many different ways of doing the same thing. In the first case, a universal guideline is this: you cannot know how you are going to feel about a set of key strokes after you have used them a thousand times, until you have actually used them a thousand times. Be empirical. A list of editors that were tested is: - Borland C IDE - jed (Unix terminal editor) - ncedit.exe (internal editor of the Norton Commander) - ne.com (Norton Editor for DOS) - notepad.exe (Windows 3.1 editor) - tvedit.exe (Turbo Vision Borland editor class for DOS) Each of these were used to write many thousands of lines of code. Others were used, but not as intensively. A grievous omission was the failure to test Emacs, however at the time, Emacs was thought to not make full use of the potential of the X keyboard standard and, like Vi, was not considered in the same spirit of user friendliness and ergonomics. Cooledit wanted to eliminated double key combinations as much as possible, as well as eliminate any sort of learning curve for novice computer users. In the second case, it is desirable to implement three separate methods of invoking a feature. First, a mouse can be used to pull the appropriate menu. Second, a hotkey should be supported, and finally, keys can be used to navigate to the menu and manually invoke the menu item. Development has aimed to make Cooledit completely independent of mouse operations, allowing it to retain the speed of a terminal editor. Having three methods of actuating a function inherently supports the users learning curve toward fluent use of the application. It is felt that all GUI's should intrinsically support this. For interest, Cooledit development began under jed. As soon as it was complete enough to save an edit buffer, jed was discarded. Thereafter, Cooledit was used to write itself. In this way it receives ongoing and intensive testing. Cooledit's interface has evolved only out of an enormous number of hours of testing and fine tuning. EDIT BUFFER DESIGN ------------------ Most editors use a single linear memory block for the edit buffer. Cooledit however has implemented a buffer array of 64k blocks. The exact details of this are explained in the sources. This buffer array system could allow data to be saved to swap files when very large files are being edited, and allow for a 16 bit implementation of the editor. However, this was never implemented, and hence the buffer system remains an odd implementation. Low level access to the buffer occurs solely through six functions: edit_get_byte - to retrieve a character. edit_insert - insert at the cursor and move one place. edit_insert_ahead - insert at the cursor. edit_delete - delete ahead. edit_backspace - delete backward. edit_cursor_move - move the cursor an integer number of places. These functions record each modification into a wrapping history - the undo stack. Hence each action taken on the buffer is recorded. This allows for the key-for-key undo feature of Cooledit. The undo stack can be set to an arbitrary size. The buffer is eight bit clean and null transparent so that binary files can be editing flawlessly. An important consideration is that no changes are made to the buffer unless it is explicitely modified by the user. Hence loading a file, moving the cursor and saving it again, leaves it completely unchanged, even if it is a binary file. (Some DOS editors do not have this grace.) DISPLAY OPTIMIZATION -------------------- Another reason for the development of Cooledit, was that other editors were not display optimized. Any application that uses a canvas area should allow for use of non-acclerated graphics displays. This requires that only the minimal surface area is redrawn with each key press. This has become less important today, as accelerated hardware becomes cheaper. Cooledit also supports 16 color displays extremely well. To demonstrate this, Cooledit has the hidden key combination Ctrl-Alt-Shift-~ which blanks the display in red. It will be noticed that moving and scrolling redraws only the minimal amount of area. A static cache is used to perform this optimization. It holds a row/column array of characters and their respective foreground and background colorizations. Redraws are compared against the cache similar to the way that xterms do. The result of this is that Cooledit uses more CPU than most other editors for display. On the other hand, X CPU usage is very much lower. GDB AND SHELL INTERACTION ------------------------- The Coolwidget library properly monitors jobs in the way of a shell. It has the facility to add call-backs to file descriptors, watching its X connection at the same time. The entire application centres around a single select() statement. Builtin is a utility function called triple_pipe_open used for any kind of process interaction. The function forks a process analogous to popen, but allows reading from stdout and stderr and writing to stdin. This function is straight forward use of process forking and file descriptor manipulation but is worth noting because novice C programmers often require this functionality, but are too inexperienced to implement it themselves or to dig for existing implementations. (Details for those wishing to use the code may be found in the sources). The interface to gdb uses this Coolwidget's call-back mechanism. It operates completely asynchronously - commands sent to gdb are queued for writing so as not to interfere with normal editor operations, and each queued command is paired with a response. The gdb interface tries to be as transparent as possible, giving the impression of a builtin debugger. It implements sufficient features that a user should rarely have to type in a command manually. The debugger allows program output to be displayed to an xterm. Gdb has the command-line option to set the tty to use for program output. However there is no reliable method of returning the tty name of the xterm back to the debugger. A small program, ttyname_stop, is installed with Cooledit to solve this problem. Xterm is run with `-e ttyname_stop'. Ttyname_stop prints its pid and controlling terminal (from the ttyname() system call) to a temporary pipe file created in the user's home directory (which is then read by the debugger) whereupon ttyname_stop pauses indefinitely. Program output is then neatly sent to the xterm. To close the xterm, the debugger merely needs to send SIGTERM to ttyname_stop's pid. Having a debugger and editor combined is most convenient for the user. With the advent of this feature, Cooledit brings the convenience of the famed Borland interfaces to Unix. SYNTAX HIGHLIGHTING ------------------- Conventional syntax highlighting buffers a color for each character (or group of characters), so that particular regions of the edit buffer are painted from `memory'. The cached colors are updated when modifications are made to the text. However, the syntax highlighting used here looks up the color of the text from its context on-the-fly, that is, as it is being drawn. Color information for a character is NOT buffered anywhere. To do this quickly requires a carefully optimized algorithm. It works by stepping forward through the text and switching modes depending on whether it has found the boundary of a word or a different keyword set. Speed is assisted by caching the first letter of each keyword. Highlighting text in this way is a novel approach, but has certain limitations - it would be to slow to support regular expressions. It also does not support case insensitive keywords. It has the advantage that there is never any highlighting `lag'. The algorithm is also transparent to line breaks. Hence, for example, a C style quote or comment can cross many lines and will be highlighted correctly. The code parses most program text at over 300kB per second on a PII300. Screen refreshes are therefore instantaneous in most circumstances. ON-THE-FLY SPELL CHECKING ------------------------- It is interesting that this is an extremely easy feature to add. Ispell allows for interaction with other programs using its `-a' option. The author of ispell may have intended for this to be used from within dialogs, say from a `Spellcheck' menu option. However, on a fast enough system, there is no reason why words cannot be continuously fed to ispell. It is in fact so easy to implement, that the author encourages every other interactive application that processes text in any way whatsoever, to add this feature. Spell-checking within program comments is also implemented. The spell-check code merely looks if the `spellcheck' option has been enabled for the particular syntax highlighting context. Outside of say, comments and string constants, spell-checking is disabled. The main problem with spell checking had most to do with how misspelled words were underlined. Either proper text styles (used by other editors) needed to be implemented, or else Cooledit had to use the existing syntax code. The later approach was considered more expedient. Misspelled words are dynamically added to the list of keywords in the syntax rule set, but are underlined instead of colorized. This has the interesting effect that a misspelled word will be highlighted everywhere in the buffer so long as your cursor has passed over it at least once. To prevent an excess of keywords (this would slow down the editor) keywords are removed from the rule set after aging one minute - which has the benefit of preventing the display from becoming cluttered with too many misspelled words. The one deficiency of Cooledit's ispell interaction is that it is not asynchronous. It has not been tested with very large ispell dictionaries or on slow machines, so this may be an area worth optimizing. (On-the-fly spell checking was defiantly added after the author read a quote by a prominent figure in the computing world, that Unix systems did not have spell-check-as-you-type.) BUILTIN PYTHON INTERPRETOR -------------------------- Python features standard procedures for building itself into high level applications, and creating interfaces to C functions. Cooledit can be built without the Python interpretor, but then lacks any Python extensions - this is mostly for non-Linux systems, where administrators do not wish to install the large Python sources. Besides basic editor operations, wrappers were created to allow users to access some of the Coolwidget library. Dialog boxes can be created from within Python in the same style as the rest of the application. In this way, Cooledit can actually be used to create simple GUI applications. Users also have the benefit of being able to add or remove from any of the existing menus, and can hence program any kind of customization. The Python interpretor is initialized on startup, and runs the script lib/cooledit/global.py and ~/.cedit/global.py. One of these scripts must define a function type_change(s), which is run whenever a file is opened or its `type' (meaning the syntax highlighting rule set being used) is changed. `s' is the string you would normally see displayed on the left of the editor window which describes the programming language: it is taken from the syntax definitions. The function can use this to decide what new utilities to make available. The C extensions currently look like this: def type_change(s): menu ("Util") # clear the Util menu # add new stuff to the util menu: if s == "C/C++ Program": menu ("Util", "for(;;) {", "c_generic('for (;;) {', 5)") menu ("Util", "while() {", "c_generic('while () {', 7)") menu ("Util", "do {", "c_do_while()") menu ("Util", "switch() {", "c_generic('switch () {', 8)") menu ("Util", "case:", "c_case()") menu ("Util", "if() {", "c_generic('if () {', 4)") menu ("Util", "main() {", "c_main()") menu ("Util", "#include ", "c_include()") menu ("Util", "printf();", "c_printf()") Where a function like c_printf() is defined further above. The C example shows how utilities may typically be coded, and serves as a tutorial. Basic operations like moving through the buffer, editing the buffer, and returning status information are provided. The Python wrappers make liberal use of Python's optional argument feature. For example, the get_line() function returns: the current line with no arguments, a single line with one argument, and a range of lines with two arguments. FUTURE DIRECTIONS ----------------- The potential that Python gives Cooledit is enormous. It is hoped that users will contribute Python utilities just as they did syntax rule sets. The Python interpretor will allow Cooledit development to subside somewhat. The C side of Cooledit is considered substantially complete. Some users have asked for a Gtk interface to Cooledit. A full Gtk interface is improbable because of the heavy reliance Cooledit makes on the Coolwidget library. The Midnight Commander for Gnome does however implement a minimal version of Cooledit under Gtk, and it is hoped that this will be extended to offer more features.