TextTokenizer Class Reference

Simple text tokenizer. More...

Collaboration diagram for TextTokenizer:
[legend]

List of all members.

Public Types

enum  Features { WANT_NEWLINE = 0x01, WANT_NEWLINE = 0x01 }
enum  Features { WANT_NEWLINE = 0x01, WANT_NEWLINE = 0x01 }
enum  TokenType {
  INVALID = -1, END, PUNCT, IDENTIFIER,
  STRING, DOUBLE, INTEGER, INVALID = -1,
  END, PUNCT, IDENTIFIER, STRING,
  DOUBLE, INTEGER
}
enum  TokenType {
  INVALID = -1, END, PUNCT, IDENTIFIER,
  STRING, DOUBLE, INTEGER, INVALID = -1,
  END, PUNCT, IDENTIFIER, STRING,
  DOUBLE, INTEGER
}

Public Member Functions

const String & Current () const
const String & Current () const
TokenType CurrentType () const
TokenType CurrentType () const
void Error (const char *msg,...)
void Error (const char *msg,...)
void Gobble ()
void Gobble ()
int line () const
int line () const
bool Match (const char *exact)
bool Match (const char *exact)
bool MatchDouble ()
bool MatchDouble ()
bool MatchIdentifier ()
bool MatchIdentifier ()
bool MatchInteger ()
bool MatchInteger ()
bool MatchNoCase (const char *str)
bool MatchNoCase (const char *str)
bool MatchNumber ()
bool MatchNumber ()
bool MatchPunct ()
bool MatchPunct ()
bool MatchQuasiIdentifier ()
bool MatchQuasiIdentifier ()
bool MatchString ()
bool MatchString ()
bool MatchType (TokenType type)
bool MatchType (TokenType type)
bool MoreTokens () const
bool MoreTokens () const
success_t Open (const char *fname, const char *comment_chars="", const char *ident_extra="", int features=0)
success_t Open (const char *fname, const char *comment_chars="", const char *ident_extra="", int features=0)
const String & Peek () const
const String & Peek () const
TokenType PeekType () const
TokenType PeekType () const

Detailed Description

Simple text tokenizer.

This tokenizes the input stream. It will identify the general type of token, and ignore whitespace and different kinds of comments.

This has the concept of 'next' token which you can peek at, and the current token. The current token is always initialized to empty at the very beginning, because you are encouraged to peek ahead one token.

Example:

 TextTokenizer tokenizer;
 tokenizer.Open("file.txt");
 
 if (tokenizer->Match("count")) {
   if (tokenizer->Match(TextTokenizer::INTEGER)) {
     printf("Found number: %d\n", atoi(tokenizer->Current()));
   } else { Error(); }
 } else { Error(); }

Definition at line 169 of file textfile.h.

The documentation for this class was generated from the following files:
Generated on Mon Jan 24 12:04:40 2011 for FASTlib by  doxygen 1.6.3