Building Parsers With Javaв„ў

By Steven  John  Metsker

The TokenStringSource class in sjm.parse.tokens is a utility that helps divide an input stream into lines that a parser can parse one at a time. Figure 9.16 shows this class.

This class provides an enumeration over the tokens of a tokenizer. Each call to nextTokenString() results in a TokenString that contains all the tokens up to a specified delimiter. This is useful for languages that consist of statements that are separated by a delimiter, such as " ; ". By using a TokenStringSource , the author of a parser need only parse individual statements of the language. The main() method of TokenStringSource gives the following example:

public static void main(String args[]) { String s = "I came; I saw; I left in peace;"; TokenStringSource tss = new TokenStringSource(new Tokenizer(s), ";"); while (tss.hasMoreTokenStrings()) { System.out.println(tss.nextTokenString()); } }

This prints the following:

I came I saw I left in peace

The Logikus parser in Chapter 14, "Parsing a Logic Language," uses TokenStringSource to break up lines of input. Statements in a Logikus program are rules and facts, separated by semicolons. The Logikus parser does not parse an entire program. Instead, it parses a single statement and relies on a TokenStringSource object to divide the input into statements. Using TokenStringSource can make your parser easier to write and faster, too.