String::Tokenizer 0.05

String::Tokenizer is a simple string tokenizer. SYNOPSIS use String::Tokenizer; # create the tokenizer and tokenize input my $tokenizer = String::Tokenizer->new("((5+5) * 10)", +*()); # create tokenizer my $tokenizer = String::Tokenizer->new(); # ... then tokenize the string $tokenizer->tokenize("((5 + 5) - 10)", ()); # will print (, (, 5, +, 5, ), -, 10, ) print join ", " => $tokenizer->getTokens(); # create tokenizer which retains whitespace my $st = String::Tokenizer->new( this is a test with, (signifigant) whitespace, ,(), String::Tokenizer->RETAIN_WHITESPACE ); # this will print: # this, , is, , a, , test, , with, , (, signifigant, ), , whitespace print "" . (join ", " => $tokenizer->getTokens()) . ""; # get a token iterator my $i = $tokenizer->iterator(); while ($i->hasNextToken()) { my $next = $i->nextToken(); # peek ahead at the next token my $look_ahead = $i->lookAheadToken(); # ... # skip the next 2 tokens $i->skipTokens(2); # ... # then backtrack 1 token my $previous = $i->prevToken(); # ... # get the current token my $current = $i->currentToken(); # ... } A simple string tokenizer which takes a string and splits it on whitespace. It also optionally takes a string of characters to use as delimiters, and returns them with the token set as well. This allows for splitting the string in many different ways. This is a very basic tokenizer, so more complex needs should be either addressed with a custom written tokenizer or post-processing of the output generated by this module. Basically, this will not fill everyones needs, but it spans a gap between simple split / /, $string and the other options that involve much larger and complex modules. Also note that this is not a lexical analyser. Many people confuse tokenization with lexical analysis. A tokenizer mearly splits its input into specific chunks, a lexical analyzer classifies those chunks. Sometimes these two steps are combined, but not here..

Requirements:	No special requirements
Platforms:	Linux
Keyword:	I-,  Libraries,  Programming,  Simple,  String,  Stringtokenizer,  Token,  Tokenizer,  Whitespace
Users rating:	0/10