Interval Research Corporation

1801 Page Mill Road, Building C

Palo Alto, CA 94304, USA

Tel: 1-650-842-6264

E-mail: {lewis, weyers}@interval.com

Most application currently used for creating text treat it as a fundamentally static medium. Dynamics and interactivity can be added to it, but only with immense difficulty. The tools normally used are complex and expensive. Substantial training is required to use them well. These drawbacks could be ignored as long as the greatest consumers of such tools were based in high-cost industries like film and video production.

The creation of a digital document that includes static, dynamic and interactive text is divided into several discrete stages. A user must compose the text, lay it out, manipulate the visual form at both gross and small levels of detail, apply dynamic behaviors and adjust the parameters of those behaviors, and specify any desired interactivity. We created the ActiveText architecture to make this process as fluid as possible.

Imagine that a user wants to make a short text-based piece which deals with how gossip can corrode the cohesiveness of a community. The user's idea is to have a single word, "gossip", affect a larger passage of text one word at a time. "Gossip" will move around the screen slowly, vibrating with excitement to convey the busy-body nature of gossips. When other words come into contact with "gossip", they will become "infected" with the same behavior, including the ability to spread their own behaviors further. Figure 1 shows several frames of what such a piece might look like.

If the user has an ActiveText-based application like the It's Alive! prototype, the process would be as follows. First the user composes, edits and lays out the words on the screen. The user then selects the word "gossip" and applies the cruise behavior to it. A palette of properties particular to that behavior pops up, allowing the user to adjust the speed and the degree of directional randomness with which the word will move about the screen.

Next the user selects all of the individual glyphs of the word "gossip" and applies the vibrate behavior to them. Another palette pops up with parameters to modify the frequency and degree of vibration. (We use the term "glyph" as a general term for any marking, symbol or character,) Finally, the user re-selects the entire word. The viral behavior is applied at this level, instructing "gossip" to transfer all of its behaviors to whatever other text elements it contacts.

The user saves the file and then switches It's Alive! into simulate mode. "Gossip" wanders around for a bit and then hits a large body of text. Its behaviors start spreading from word to word and glyph to glyph. The text block slowly breaks apart until all of its words are wandering around the screen, vibrating.

At this point, the user realizes that that there are some spelling errors in the larger block of text, and that there is no title to the piece. The user can choose to copyedit while the document is in action, or by turning the simulation off. The user chooses the latter, loading the saved file to return to the original composition. The user also adds the title—"corrosion"—at the bottom left of the screen and applies the immunity behavior to it. This will keep the title from being affected by the viral behavior, and thus it will remain in place.

Creating such a document with conventional tools would be a significantly more involved process. Current tools enforce a degree of compartmentalization that reflects the many stages enumerated above. Creating a document like the "gossip" piece requires that the user work first with Microsoft Word [1], then move to Quark Xpress [2], then to Photoshop, then finally to Adobe AfterEffects [3] and/or Macromedia Director [4]. Making the process more onerous, the user can move text data through the tool chain in the direction described without too much difficulty. Moving them in the other direction, from Photoshop to Xpress, for instance, is hard if not impossible.

Applying behaviors such as those in the "gossip" piece would be even more time-consuming. If the user wanted to make an animation, each individual glyph would have to be hand-animated. The complexity of interaction between text objects as they come into contact with and infect one another would be challenging task a for a seasoned professional.

If the user wanted to circumvent the involved animating process, she could use Director to script the desired behaviors. This approach suffers from several deficiencies. Coordinating the vibrate behavior at the glyph-level with the cruise behavior at the word-level would require a custom object model. Even after custom-crafting this object model, the user would find it limited. It would not support multiple inheritance or polymorphism, both of which make managing the behaviors of objects within objects more tractable. The interpreter would struggle to deliver satisfactory real-time performance as the complexity of the object model grew. At run-time Director converts display-text to pixel-only representations, removing the possibility of copy-editing while the document is in action.

The difficulty in applying complex behaviors, the multiplicity of tools and the asymmetry between them conspire to make rapid prototyping of textural appearance, dynamic behavior and user interaction extremely difficult for all but the most experienced users. Such a process wastes time, stifles creativity, and discourages experimentation.

Figure 2: The location of the ACII-pixel wall in the flow of work required to make a dynamic and interactive text.

In the context where every byte of memory is precious, throwing away information in this manner is necessary to conserve storage space. The memory capacities of modern personal computers mitigate the need to be so conservative. If we retain such information not only can we move easily between ASCII and pixels, but we can also build upon it to offer other capabilities which are based on the syntactic structure of the text. We can cause all of the nouns in a sentence to vibrate, or all of the verbs to wriggle. We can perform dictionary and thesaurus look-ups and make substitutions on the fly. Or we can have all the articles in a sentence cluster into one corner of the screen as if shirking their yeoman's duties.

A common means of internally representing text is by parsing the text into a one-dimensional array. The resulting "stream" of characters is manipulated via an offset into the stream that determines where characters are added or deleted, what font starts and stops where, what size starts and stops where, etc. The only text object the application must maintain is the stream. This method works well for a static paradigm in which the two most important pieces of information about the text are the ASCII-value of each of the characters in the stream and the font (including face, size, and style.)

The stream method is not as well-suited for a dynamic paradigm in which different chunks of text are subject to behaviors and in which the visual rendering of a character changes from moment to moment. Among other difficulties, it would require heavy modification so as to support storage for and real-time updating of the pixel representations for individual characters. It would also be a challenge to manage the interaction of various behaviors, as the stream would require constant reparsing to find the relevant chunks.

ActiveText addresses these issues by employing an object hierarchy (Figure 3.) The text is parsed into meaningful chunks such as glyph, word, passage, and text objects. Each object can be manipulated both autonomously and by inheritance from the larger structures above it in the hierarchy. Each glyph stores both its ASCII and pixel values. Behaviors mediate for control at the appropriate object level. Everything in the architecture, including objects and behaviors, is agnostic about how the glyphs are actually rendered. This encapsulation enables the development of fonts which are dynamic and interactive in and of themselves. We call these fonts "SoftType" fonts.

Figure 3: The hierarchy of text objects.

Current tools for doing dynamic and interactive text documents require the user to author in one mode and then switch to a separate mode to view the results. ActiveText supports continuous editing: all of the tools in It's Alive!, for instance, can be used for both static and dynamic composition.

On the level of innovative glyph representations one must turn to Maeda, who has a long history of exploring the dynamic aspects of letterforms [11]. Rosenberger [12] developed a font that responds to human vocal prosody by modifying the visual aspect of individual glyphs, while Cho [13] created a series of malleable and animated fonts. Haeberli has created a tool for dynamically altering the shape and look of individual three-dimensional letterforms (personal communication.). All of this work, however, is sealed into its own specialized applications. Using any of it in an full-featured dynamic word processing environment would require extensive rewriting. The SoftType component of the present work seeks to make such radical glyph representations readily available and usable in a complete writing environment.

Lewis' work on WordNozzle [14] makes a analogy between stream representations of text and streams of paint applied, grafitti-like, to a canvas. This work highlighted some of the limitations of working within the stream paradigm, particularly when attempting to apply behaviors to a large body of text.

MotionPhone [15] by Snibbe removes the distinction between authoring and performance modes. The result is a fluid, accessible tool which is particularly well-suited to experimentation but proves quite difficult to use as a composition tool. This difficulty stems from the fact that the user is forced to constantly "catch up" to his own document. The continuous editing approach taken in the present work addresses this issue by giving the user the option to edit in both static and dynamic modes.

Several commercial products can be used for creating dynamic texts. At the high-end are film effects system such as Discrete Logic's Flame and related systems [16], and Quantel's Harry[17]. These systems run on specialized hardware and require extensive customization to use fluidly. The documents they produce are not interactive. Adobe AfterEffects, which dominates the middle-range, produces non-interactive video loops. At the low end are products like Microsoft's Liquid Motion and Macromedia's Flash, with very limited interactivity and weak scripting environments.

ActiveText is a general purpose architecture for creating dynamic texts. It consists of a set of C++ libraries written on top of the Microsoft Foundation Classes [18]. The architecture parses text into an object-hierarchy based on textual granularity. Character information is stored as both ASCII- and pixel-values. Behaviors for transforming the text objects utilize a message subscription model to coordinate between and mediate among themselves. Object encapsulation enables continuous editing.

The architecture meets several challenges. As a general solution, it has to support dynamic and interactive behaviors of all forms. The text objects on which these behaviors are operating must interact with a particular behavior at the level or levels which is/are appropriate. For instance, if the user places the move to mouse behavior on individual glyphs, the interaction between glyph objects and behavior must reflect this by moving the glyphs independently towards the mouse. If the user places move to mouse on a word objects, the appropriate result is to have the word move as a unit towards the mouse.

Text objects must have a means of notifying a behavior if it is not a candidate for the behavior's particular function. An example of such a case is the synonym shift behavior, which uses the WordNet [19] lexicon to cycle a word through its different synonyms. A synonym is a word-level phenomenon; it makes no sense to apply the concept to a glyph. Yet it does make sense to apply it to a passage if the user wants to have all of the words in a passage cycle through their synonyms. The architecture must support such nuance.

In order to support complex, layered behaviors, the architecture must provide a means to negotiate control of a particular object. It must also support the activation and deactivation of behaviors by other behaviors. Similarly, support for SoftType fonts requires that the architecture de-couple how behaviors operate on text elements from how the actual glyphs themselves are drawn.

Finally, the continuous editing requirement means that all of this coordination must be possible while the document is in motion and the user is interacting with it. If the user deletes an object, wherever it is in the hierarchy, the architecture must remove it cleanly and notify behaviors which are acting upon it of this removal. If objects are added, such as adding a word object to a passage object, the behaviors must be able to incorporate the new object into their actions on the fly. Further complicating matters is the fact that not only can the user add elements and behaviors and adjust their properties, but other behaviors can do this as well.

The basic element in the ActiveText data structure is the book object. The book functions as both an encapsulation of the text objects and an engine for driving behaviors. Every new ActiveText document creates a single book, and it is the book that is stored and retrieved when a file is saved or loaded.

Text, passage, word and glyph objects are all derived from the same base object class. This class maintains pointers to the other members in the object hierarchy. Each object inherits a set of basic properties that includes font, color, and position. The positions of all objects are relative to that object’s parent. Consequently, a glyph stores its location as a position within the coordinate system of its parent word. The position of a word is an offset from its parent passage. This arrangement facilitates the sensible positioning of objects at different levels within the hierarchy, either by the user or by behaviors that need to naively operate on objects regardless of their type.

The encapsulation of the objects in the hierarchy allows ActiveText to support continuous editing, i.e., the composition, layout, and creation of complex behaviors while the document is in action.

ActiveText behaviors are objects that alter the properties of text objects over time or in response to user interactions. Such properties include font, color and position. Each behavior has a list of subjects (i.e., text objects) upon which the behavior acts (Figure 4.) Both user interaction and the activity of other behaviors can modify the subject list. Behaviors and objects are coordinated using a message subscription model [20].

When a behavior is instantiated by either the user or by another behavior, the instance is added to the book's list of behavior objects. Initially, behaviors are inactive. At the point of activation they subscribe to the appropriate messages. The book will then notify that behavior instance when an event of interest occurs. Similarly when a behavior is deactivated, its instance is unsubscribed from all of the messages with which it had registered.

Behaviors are capable of creating new properties on members of their subject list. These properties can be accessed by behaviors other than the one which created them. Behaviors which operate on the same properties do so successively (Figure 5.)

By creating, deleting or reordering objects, behaviors can manipulate the gross structure of the text. They can also create or destroy other instances of behaviors.

In order to safely mix standard TrueType fonts and SoftType fonts, the rendering of glyph outlines is performed by an ElementRenderer object (Figure 7.) The ElementRenderer determines how and when the object will draw itself on-screen. An object without an ElementRenderer will not draw itself. The first time a text is entered and parsed into objects, each glyph is given an ElementRenderer set to the default or standard text output. Words and passages do not have an ElementRenderer unless the user or another behavior explicitly assigns one.

(a)

(b)

Like behaviors, ElementRenderers are capable of subscribing to messages. They can be applied at any level in the hierarchy. Unlike behaviors, an object has at most one ElementRenderer. This restriction is to prevent the needless activity that would occur if multiple ElementRenderers attempted to draw the same object.

Two types of pop-up windows, or palettes, are used. The properties palette displays the properties of a selected object. The behavior palette displays a specific behavior and provides access to the parameters of that behavior. Following Bier, et al, [22] the palettes have variable transparency. This allows the user to make adjustments while maintaining view of the full canvas.

In order to keep track of what text objects are related to each other and how, It's Alive! uses a node-and-spoke display. An example can be seen in the middle of Figure 8. Small circles are placed under each glyph in a word. Spokes connect each glyph to a medium circle placed under the center of the word. This circle is in turn connected to a large circle which is at the center of the passage to which it belongs. All of the passage are connected by spokes to the largest circle, which is placed at the center of the entire text. This display is called a "spider" display within the document. With it the user can see which glyphs are related to which words, which words to which passages, etc. This is particularly helpful for dynamic sketching, or while editing complex documents.

Environment, property and behavior functionality is accessed through a pie menu system derived from Hopkins [21]. The pie menu is activated by the right mouse button (see lower-left of Figure 8.). It has four axes: typographic, selection, behaviors, and file. Each of these axes opens out into a series of submenus:

Behaviors The user uses this menu to apply or remove

behaviors to and from selected text objects (Figure 9.). This is also where the user can choose to turn the dynamics on or off by selecting "simulation".

File Operations such as creating a new document, loading a file, QuickTime output and other exporting functionality are accessed here.

Typographic Controls the appearance of glyphs in the text (Figure 10.) The user can change the font, size, color and style from this menu. All of these submenus provide instant feedback, e.g., as the user moves the mouse through the list of fonts the font of the selected text changes appropriately.

We conducted a user study on a highly restricted set of the ActiveText functionality embodied in It's Alive! The subjects' exposure to the full capabilities of the prototype was limited in order to focus on how they reacted to behaviors, and, in particular, if they thought it possible that they would make use of such behaviors in their daily text-based communication. The It's Alive! interface in all its complexity was not tested. For most task, users button-clicked through different examples of simple documents made with It's Alive! In one task they were themselves able to add, remove and layer behaviors.

The subject pool was composed of eight dyads of experienced personal computer users. All were from a nearby undergraduate and graduate university. Subject ages ranged from 20 to 24 years old, with an equal number of males and females.

Preliminary results from the test suggest that users found the behaviors interesting and novel, but it is unclear how much they might use them in every day communication. Users showed the most interested when layered behaviors produced unexpected results, such as adding a wave motion to an explode behavior, or by combining any of the SoftType fonts with anything else. The subjects clearly thought it necessary that the application of behaviors require little time or overhead, particularly for communication media such as chat and email. We look forward to testing the full It's Alive! interface to see if it is lightweight enough to address their concerns.

The subjects' responses were not the only valuable outcome of the study. No crashes happened over a week of rigorous testing, suggesting that the ActiveText libraries underlying It's Alive! possess a fair degree of robustness.

A simple timeline allows the users to specify when objects appear on screen, when behaviors turn on or off, and get an overview of longer documents. An environment-level behavior called ActivePaper has been defined. ActivePapers are regions of the screen which interact in a particular way with whatever text passes through them. The incorporation of a Postscript module for exporting individual glyphs makes It's Alive! available for use as a tool for experimenting with typography.

As it is currently written, ActiveText anticipates run-time registration of behaviors and SoftType fonts. Except for text objects, all the critical objects in the system possesses a unique identifier. These identifiers are declared statically in the current version of It's Alive!. In the next version these identifiers will be generated at run-time as part of registration process. Run-time registration will allow third-parties to create plug-in behaviors and SoftType fonts.

The ActiveText architecture improves the process of creating a dynamic and interactive text. It releases the user from having to worry about crossing over the ASCII-pixel wall and not being able to return. It permits the user to both compose statically and sketch dynamically. The architecture supports increasingly complex applications of behaviors, and provides a robust means for incorporating radical glyph representations such as those represented by the SoftType fonts.

Arriving at the present implementation involved significant experimentation and learning. From that process, we have selected a few points that may be of particular interest to anyone interested in utilizing the existing libraries or implementing a similar architecture.

Designing a tractable hierarchy is not the same as designing a meaningful one. The composition of the object hierarchy above the level of the word and below the level of the text, i.e., at the level we presently call the "passage", is questionable. We considered several other chunks at this level, including sentences and phrases. "Passage" proved to be general enough to encompass both terms. Yet it denotes a vagueness which is unsatisfying, and which does not reflect the way in which people talk about language. We anticipate user studies will assist us on resolving this matter.

Complex interactions among complex behaviors frustrates predictability. Once behaviors reach a sufficient degree of complexity, it is difficult to predict the outcome of their interaction. Seen one way, this provides a constant stream of pleasant surprises on the order of "that's interesting; how did that happen?" and is a fruitful way of exploring unexpected compositions. Seen another way, this lack of predictability leads to frustration as the text behaves in undesirable ways. One solution to the predictability issue is to rigorously divide behaviors into a set of primitive behaviors and a set of compound behaviors, with the former built upon the latter. The primitive behaviors would be simple enough that the user could reliable predict what certain combinations of them will be like. Another solution may lie at the interface, in presenting behaviors to the user in functional groupings and in providing a quick preview mechanism.

We have argued that current tools for creating dynamic and interactive texts limit the ability of the user to conceive of, experiment with and produce such texts. The proposed ActiveText architecture supports a more expressive and fluid method for creating such texts. The object-based hierarchy, combined with dual ASCII-pixel representations, a mechanism for sensibly applying and coordinating behaviors, and rigid data encapsulation allow the user to easily modify the text in interesting way in both static and dynamic modes. We have described a prototype, called It's Alive!, which is built on top of ActiveText libraries. Finally, we have reported cautiously positive results from a user-experience study in which subjects were exposed to a limited portion of the It's Alive! application.

We anticipate that the next major phase of research will address issues such as:

Refining the behavior model. We need to refine the behavior model to allow conflicting behaviors to negotiate for control of text object properties.

Integration into on-line applications. We would like to investigate ways of making the ActiveText functionality accessible to browsers, email applications and chat. This will most likely be done by constructing the appropriate plug-ins, but may also be accomplished within a Java or modified Flash framework.

Performance version. We also wish to develop a version suitable to live performance of spoken word, perhaps incorporating functionality similar to Rosenberger's Prosodic Font.

Further iterations on the user interface. We would like to further refine the user interface, and test it to see if it is indeed an improvement over the standard toolbar interface found in conventional text-tools.

Integrating computational linguistics. The text's existence as a fully parsed structure provides easy access to words and passages/phrases/sentences. The addition of a dynamic parts-of-speech tagger would enable users to specify behavior based upon whether a word is a noun or a verb. Combined with lexical information similar to that provided by WordNet, a wealth of potentially interesting behaviors operating simultaneously on semantics and aesthetics becomes possible.

Scott Snibbe and Douglas Soo assisted in the develolpment SoftType concept, and conversations with Sha Xin Wei led to the ActivePaper concept. Jenny Dana and Tom Ngo helped with the intricacies of mediating behaviors. Warren Sack provided sound advice on computational linguistics, and Gavin Miller lent optimization assistance. Debby Hindus provided invaluable commentary on several drafts of this paper.

2. Quark Corporation. Product specifications. On-line at http://www.quark.com/
3. Adobe Corporation. Product specifications. On-line at http://www.adobe.com/.
4. Macromedia Corporation. Product Specifications. On-line at http://www.macromedia.com.
5. Ishizaki, Suguru. Multiagent Model of Dynamic Design: Visualization as an Emergent Behavior of Active Design Agent. In Proceedings of CHI '96 Human Factors in Computing Systems, pp. 347 - 354, Vancouver, BC, April 1996.
6. Wong, Yin Yin. Temporal Typography: Characterization of time-varying typographic forms. Masters thesis, Massachusetts Institute of Technology, 1995.
7. Soo, Douglas. Implementation of a temporal typography system. Masters thesis, Massachusetts Institute of Technology, 1997.
8. Chao, Chloe. Kinetext: A concrete programming paradigm for kinetic typography. Masters thesis, Massachusetts Institute of Technology, 1998.
9. David Small. Navigating large bodies of text. IBM Systems Journal, Vol. 35, No. 3&4, 1996.
10. David Small and David Small. An Interactive Poetic Garden. In Proceedings of CHI '98 Human Factors in Computing Systems, pp. 303-304, LA, CA, April 18-23, 1998.
11. John Maeda. Flying Letters. Reactive Book Series, Number Two. Digitalogue, Tokyo, 1996.
12. Rosenberger, Tara. Prosodic Font: The space between the spoken and the written. Masters thesis, Massachusetts Institute for Technology, 1998.
13. Cho, Peter. Pliant Type: Development and temporal manipulation of expressive, malleable typography. Masters thesis, Massachusetts Institute for Technology, 1997.
14. Lewis, Jason. WordNozzle: Painting with Words. Sketches section of the Special Interest Group Graphics (SIGGRAPH) Annual Conference, Los Angeles, August 1997.
15. Snibbe, Scott. MotionPhone. Interactive communities section of the Special Interest Group Graphics (SIGGRAPH) Annual Conference, Los Angeles, August 1995.
16. Discrete Logic. On-line at http://www.discrete.com.
17. Quantel Corporation. On-line at http://www.quantel. com.
18. Microsoft Visual Studio 6.0. Microsoft Corporation, Redmond, WA, 1998.
19. Christiane Fellbaum (ed.) WordNet: An electronic lexical database. MIT Press, Cambridge, MA, 1998.
20. Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides. Design Patterns: Elements of reusable objected oriented software. Addison-Wesley, Reading, MA, 1995.
21. Hopkins, Don. The design and implementation of pie Menus. Dr. Dobb's Journal, December, 1991, 16-26.
22. Bier, Eric A., Maureen C. Stone, Ken Pier, William Buxton, and Tony DeRose. Toolglass and Magic Lenses: the see-through interface.