Return to Index

Tokenizer

The Tokenizer class provides a simple way to split delimited strings into tokens. The tokenizer can be configured to recognize any set of single characters as delimiters. A tokenizer can be used either with the hasTokens() and next() methods, or with the more flexible get() and unget() methods. When get(..) is called, it returns the next token in a queue of tokens. If the token is not ready to be accepted, it can be returned to the queue in the correct order with the unget() method, which can be called as many times as required. This way, the list of tokenized items can be traversed back and forth if necessary.

Method Summary:
Tokenizer::Tokenizer();
Tokenizer::Tokenizer(const String& s);
Tokenizer::Tokenizer(const String& s, const String& dlms);
String Tokenizer::next();
bool Tokenizer::isEmpty();
bool Tokenizer::get(String& pTok);
bool Tokenizer::unget();
void Tokenizer::clearQueue();
bool Tokenizer::hasTokens();
int Tokenizer::getLastTokenEndPos();
int Tokenizer::getCurrentPos();
String Tokenizer::getRestOfInput();
void Tokenizer::reset(const String& s);
void Tokenizer::setDelimiters(const String& dlms);
String Tokenizer::getDelimiters();


Tokenizer::Tokenizer
Tokenizer::Tokenizer();

Creates an empty space-delimited tokenizer.
Tokenizer::Tokenizer
Tokenizer::Tokenizer(const String& s);

Creates a new space-delimited tokenizer with the given input string.
Parameters:
const String& sThe string to tokenize.
Tokenizer::Tokenizer
Tokenizer::Tokenizer(const String& s, const String& dlms);

Creates a new tokenizer on the given input string with custom delimiters.
Parameters:
const String& sThe string to tokenize.
const String& dlmsA string of delimiter characters.
String Tokenizer::next
String Tokenizer::next();

Returns the next token in the input. Use in conjunction with hasTokens().
bool Tokenizer::isEmpty
bool Tokenizer::isEmpty();

Determines whether the token queue is empty.
bool Tokenizer::get
bool Tokenizer::get(String& pTok);

Gets the next token in the queue.

Returns: Returns false if the queue was empty.
Parameters:
String& pTokThe string to assign the next token to.
bool Tokenizer::unget
bool Tokenizer::unget();

Ungets one token and puts it back on the front of the token queue.

Returns: Returns false if a token could not be put back on the queue.
void Tokenizer::clearQueue
void Tokenizer::clearQueue();

Clears the token queue.
bool Tokenizer::hasTokens
bool Tokenizer::hasTokens();

Returns whether tokens remain on the input string. Use with next().
int Tokenizer::getLastTokenEndPos
int Tokenizer::getLastTokenEndPos();

Returns the ending position of the most recently processed token in the input string.
int Tokenizer::getCurrentPos
int Tokenizer::getCurrentPos();

Returns the current position of the tokenizer in the input string.
String Tokenizer::getRestOfInput
String Tokenizer::getRestOfInput();

Returns the rest of the unprocessed input string.
void Tokenizer::reset
void Tokenizer::reset(const String& s);

Resets the tokenizer to a new input string. Note that this does not clear the token queue.
Parameters:
const String& sThe new input string.
void Tokenizer::setDelimiters
void Tokenizer::setDelimiters(const String& dlms);

Changes the character delimiters.
Parameters:
const String& dlmsA string of single character delimiters.
String Tokenizer::getDelimiters
String Tokenizer::getDelimiters();

Returns the current delimiters.
Generated automatically by docgen 0.0.1
© 2003 Aron Dobos