Download Shareware and Freeware Software for Windows, Linux, Macintosh, PDA

line Home  |  About Us  |  Link To Us  |  FAQ  |  Contact

Serving Software Downloads in 976 Categories, Downloaded 30.066.133 Times

String::Tokenizer 0.05

  Date Added: August 10, 2010  |  Visits: 707

String::Tokenizer

Report Broken Link
Printer Friendly Version


Product Homepage
Download (75 downloads)

String::Tokenizer is a simple string tokenizer. SYNOPSIS use String::Tokenizer; # create the tokenizer and tokenize input my $tokenizer = String::Tokenizer->new("((5+5) * 10)", +*()); # create tokenizer my $tokenizer = String::Tokenizer->new(); # ... then tokenize the string $tokenizer->tokenize("((5 + 5) - 10)", ()); # will print (, (, 5, +, 5, ), -, 10, ) print join ", " => $tokenizer->getTokens(); # create tokenizer which retains whitespace my $st = String::Tokenizer->new( this is a test with, (signifigant) whitespace, ,(), String::Tokenizer->RETAIN_WHITESPACE ); # this will print: # this, , is, , a, , test, , with, , (, signifigant, ), , whitespace print "" . (join ", " => $tokenizer->getTokens()) . ""; # get a token iterator my $i = $tokenizer->iterator(); while ($i->hasNextToken()) { my $next = $i->nextToken(); # peek ahead at the next token my $look_ahead = $i->lookAheadToken(); # ... # skip the next 2 tokens $i->skipTokens(2); # ... # then backtrack 1 token my $previous = $i->prevToken(); # ... # get the current token my $current = $i->currentToken(); # ... } A simple string tokenizer which takes a string and splits it on whitespace. It also optionally takes a string of characters to use as delimiters, and returns them with the token set as well. This allows for splitting the string in many different ways. This is a very basic tokenizer, so more complex needs should be either addressed with a custom written tokenizer or post-processing of the output generated by this module. Basically, this will not fill everyones needs, but it spans a gap between simple split / /, $string and the other options that involve much larger and complex modules. Also note that this is not a lexical analyser. Many people confuse tokenization with lexical analysis. A tokenizer mearly splits its input into specific chunks, a lexical analyzer classifies those chunks. Sometimes these two steps are combined, but not here..

Requirements: No special requirements
Platforms: Linux
Keyword: I- Libraries Programming Simple String Stringtokenizer Token Tokenizer Whitespace
Users rating: 0/10

License: Freeware Size: 8.19 KB
STRING::TOKENIZER RELATED
Programming  -  Mogemoge 1.0
ogemoge is a simple programming language included join token mechanism to ease writing game character relations and game events (ex. collision).
460.8 KB  
Programming  -  Simple String Conversion 1.0
Simple String Conversion works to convert any character string to the right version for C, C++, JavaScript or VisualBasic programming languages. The conversion speed epends, as is logical, on the quantity and length of the text to be converted,...
1024 B  
Libraries  -  kcurses
kcurses are some simple widgets and ui abstractions for ncurses, to handle list boxes, vi-like editing fields, mouse selections, and the like. Why? Because sometimes gui programs are more annoying than helpful. web interfaces suck, and gtk/qt...
184.32 KB  
Libraries  -  Acme::OneHundredNotOut 100
Acme::OneHundredNotOut is a raise of the bat, a tip of the hat. I have just released my 100th module to CPAN, the first time that anyone has reached that target. As some of you may know, I am getting ready to go back to college and reinvent...
14.34 KB  
Libraries  -  Bio::Tools::Run::PiseApplication::align2model 1.4
Bio::Tools::Run::PiseApplication::align2model is a Bioperl class for align2model - create a multiple alignment of sequences to an existing model. Parameters: align2model (String) run (String) Run name db (Sequence) Sequences to align...
829.44 KB  
Libraries  -  Data2XML 1.3
Data2XML is a quick JAXB-based tool for parsing simple-to-moderately-complex input data into a flat XML representation. It is meant to replace simple String.substring() operations and keyword searches in Java by an easy-to-use XML-configured...
4 MB  
Libraries  -  Memoize 1.01
Memoize - Make functions faster by trading space for time. SYNOPSIS # This is the documentation for Memoize 1.01 use Memoize; memoize(slow_function); slow_function(arguments); # Is faster than it was before This is normally all you need...
47.1 KB  
Libraries  -  Audio::Data 1.029
Audio::Data is a module for representing audio data to perl. SYNOPSIS use Audio::Data; my $audio = Audio::Data->new(rate => , ...); $audio->method(...) $audio OP ... Audio::Data represents audio data to perl in a fairly compact and...
88.06 KB  
Miscellaneous  -  Scramble Word 1.0
Scramble word script re-arrange characters of word by using simple string manipulation and Random module.
 
Development Tools  -  A simple CPPI strategy in MATLAB 1.0
In this set of files, I propose a simple CPPI (Constant Proportion Portfolio insurance) implementation. One can run the backtesting of such a strategy, playing with the parameters of the strategy such as Multiplier (Risk Exposure), or Smoothing...
4.15 MB  
NEW DOWNLOADS IN PROGRAMMING, LIBRARIES
Programming  -  FLEX-db Digital Asset Manager 3.0.9
FLEX-db - an enterprise Digital Asset Manager (DAM). It ingests and links metadata with files, creates thumbnails, and processes files using business rules. FLEX-db has a JSP client, Java app server for file input and output and an EJB metadata...
21.57 MB  
Programming  -  Libicom 0.9.0
The libicom library is a character based dynamicly linked library for Linux. It is used to remotely control the Icom IC-R8500 wide band receiver via an RS232 link. All call and return parameters to the control functions are character string based....
20.48 KB  
Programming  -  dotdesktop 0.3
Dotdesktop library provides ability to parse desktop entry file and access the information in a convenient way. Desktop entry file format is defined by freedesktop.org, it is used to describe information about an application such as the name and...
327.68 KB  
Programming  -  Cedalion for Linux 0.2.6
Cedalion is a programming language that allows its users to add new abstractions and define (and use) internal DSLs. Its innovation is in the fact that it uses projectional editing to allow the new abstractions to have no syntactic limitations.
471.04 KB  
Programming  -  libyasl 0.2
Libyasl is a C++ class library to easily realize TCP/UDP/Multicast clientsand servers in IPv4 and IPv6 environments under GNU/Linux systems.
143.36 KB  
Libraries  -  wolfSSL 3.12.0
The wolfSSL embedded SSL/TLS library is a lightweight SSL library written in ANSI standard C and targeted for embedded and RTOS environments - primarily because of its small size, speed, and feature set. It is commonly used in standard operating...
2.73 MB  
Libraries  -  EuGTK 4.8.9
Makes it easy to develop good- looking, fast, cross-platform programs that run on Linux, OS X, and Windows. Euphoria is a very fast interpreted/compiled language with straight-forward syntax. EuGTK allows programming in a clean, object-oriented...
10.68 MB  
Libraries  -  Linux User Group Library Manager 1.0
The LUG Library Manager is a project to help Linux User Groups start their own library. A LUG library is helpful to the community at large because it increases access to information, and gives everyone the opportunity to become more knowledgeable.
5.35 KB  
Libraries  -  Module::MakefilePL::Parse 0.12
Module::MakefilePL::Parse is a Perl module to parse required modules from Makefile.PL. SYNOPSIS use Module::MakefilePL::Parse; open $fh, Makefile.PL; $parser = Module::MakefilePL::Parse->new( join("", ) ); $info = $parser->required;...
8.19 KB  
Libraries  -  sqlpp 0.06
sqlpp Perl package is a SQL preprocessor. sqlpp is a conventional cpp-alike preprocessor taught to understand SQL ( PgSQL, in particular) syntax specificities. In addition to the standard #define/#ifdef/#else/#endif cohort, provides also...
10.24 KB