Lexing

• I will omit the arcane but tedious details of building a lexer

• See for example The Unix Programming Environment by Kernighan and Pike

• We will assume that the lexer returns tokens like this:

        1234            ["NUMBER", 1234]
        sqrt            ["FUNCTION", "sqrt"]
        x3              ["VAR", "x3"]
        ^               ["^"]
        **              ["^"]
        +               ["+"]
        *               ["*"]
        (               ["("]
        )               [")"]

• Notice how the lexer can recognize both ^ and ** and eliminate the distinction

  • This saves work in the parser

• Also notice that ** is lexed as a power operator, not as two multiplication signs

• We will imagine that our lexer scans the entire input immediately

  • Returns a linked list of all tokens

Next

Copyright © 2007 M. J. Dominus