perl-String-Tokenizer - A simple string tokenizer

Distribution: Mageia Cauldron
Repository: Mageia Core i586
Package name: perl-String-Tokenizer
Package version: 0.60.0
Package release: 2.mga6
Package architecture: noarch
Package type: rpm
Installed size: 44.22 KB
Download size: 26.49 KB
Official Mirror:
A simple string tokenizer which takes a string and splits it on whitespace. It also optionally takes a string of characters to use as delimiters, and returns them with the token set as well. This allows for splitting the string in many different ways. This is a very basic tokenizer, so more complex needs should be either addressed with a custom written tokenizer or post-processing of the output generated by this module. Basically, this will not fill everyones needs, but it spans a gap between simple 'split / /, $string' and the other options that involve much larger and complex modules. Also note that this is not a lexical analyser. Many people confuse tokenization with lexical analysis. A tokenizer mearly splits its input into specific chunks, a lexical analyzer classifies those chunks. Sometimes these two steps are combined, but not here.



  • perl(String::Tokenizer) == 0.60.0
  • perl(String::Tokenizer::Iterator)
  • perl-String-Tokenizer == 0.60.0-2.mga6

    Install Howto

    1. Enable Mageia Core repository on "Install and Remove Software"
    2. Update packages list:
      # urpmi.update -a
    3. Install perl-String-Tokenizer rpm package:
      # urpmi perl-String-Tokenizer


    • /usr/lib/perl5/vendor_perl/5.22.2/String/
    • /usr/share/doc/perl-String-Tokenizer/Changes
    • /usr/share/doc/perl-String-Tokenizer/LICENSE
    • /usr/share/doc/perl-String-Tokenizer/META.json
    • /usr/share/doc/perl-String-Tokenizer/META.yml
    • /usr/share/doc/perl-String-Tokenizer/MYMETA.yml
    • /usr/share/doc/perl-String-Tokenizer/README
    • /usr/share/man/man3/String::Tokenizer.3pm.xz


    2016-06-19 - pterjan <pterjan> 0.60.0-2.mga6 + Revision: 1033435 - Rebuild for perl 5.22.2

    2016-04-15 - shlomif <shlomif> 0.60.0-1.mga6 + Revision: 1002298 - update to 0.06

    2016-02-06 - umeabot <umeabot> 0.50.0-6.mga6 + Revision: 941134 - Mageia 6 Mass Rebuild

    2014-10-15 - umeabot <umeabot> 0.50.0-5.mga5 + Revision: 739436 - Second Mageia 5 Mass Rebuild

    2014-09-16 - umeabot <umeabot> 0.50.0-4.mga5 + Revision: 686256 - Mageia 5 Mass Rebuild

    2013-10-18 - umeabot <umeabot> 0.50.0-3.mga4 + Revision: 514713 - Mageia 4 Mass Rebuild