Tokenizing

• We can get rid of that grep:

        sub tokens {
            split m{(
                     \*\* | :=  # ** or := operator 
                    |
                     [-+*/^()=]  # some other operator
                    |
                     [A-Za-z]\w* # Identifier
                    |
                     \d*\.\d+(?:[Ee]\d+)?  # Decimal number
                    |
                     \d+         # Integer
                    )
                    |
                    \s+
                   }x, shift();
        }

• (Thanks to Andy Wardley.)

Next

Copyright © 2002 M. J. Dominus