| Next | Making Programs Faster | 65 |
parse_text splits the input into a list of tokens
Then it deals with the tokens one at a time
The existing tokenizer splits C<caller> into two tokens:
C< and caller>
It puts an object representing C< onto the stack
Then when it sees caller> it pops the stack
This complication is necessary for difficult cases like A<foo B<bar> baz>
For simple cases it is overkill
Idea:
Tokenize difficult cases as before
But tokenize simple cases like C<caller> as single tokens
| Next | |
Copyright © 2003 M. J. Dominus |