Higher-Order Parsing

Lexing

Lexing is mostly a matter of simple pattern matching
We build a scanner that works its way through the input string a character at a time
It executes a state machine
When the state machine indicates that a complete token has been read, the lexer returns the token
In C, we can also use the program lex to generate the state machine
In Perl, we usually use regular expressions