JLex

Contents

• Overview
• Format of a JLex Specification
  • Regular Expression Rules
    • Test Yourself #1
  • JLex Directives
    • Test Yourself #2
  • Comments
  • States
    • Test Yourself #3
• yyline and yytext
• A Small Example
• Quick Reference Guide

Overview

                         +-----------+
JLex specification  ---> | JLex.Main | ---> Java source code
   (xxx.jlex)            +-----------+      (xxx.jlex.java)


                    +------------------+
xxx.jlex.java  ---> | jikes (or javac) | ---> Yylex.class
                    +------------------+

                    +--------+
Yylex.class-------> |  java  | ---> output of Main
Main.class--------> |        |
                    +--------+

Format of a JLex Specification

1. User code: this part of the specification will not be discussed here.
2. JLex directives: This includes macro definitions (described below). See the JLex Reference Manual for more information about this part of the specification.
3. Regular expression rules: These rules specify how to divide up the input into tokens. Each rule includes an optional state list, a regular expression, and an associated action.

Regular Expressions Rules

The state-list part of a rule will be discussed later. Ignoring state-lists for now, the form of a regular expression rule is:

         regular-expression       { action }
	       ^                    ^
	       |                    |
	    the pattern to	  java code to be executed
	    be matched	          when the pattern is matched

When the scanner's next_token method is called, it repeats:

1. Find the longest sequence of characters in the input (starting with the current character) that matches a pattern.
2. Perform the associated action.

If an input character is not matched in any pattern, the scanner throws an exception (so it is important to make sure that there can be no such unmatched characters, since it is not good to have a scanner that can "crash" on bad input).

The regular expressions are similar to the ones discussed in the scanner notes. Here's how they are used to match the input:

• Most characters match themselves. For example:

         abc
         ==
         while
         

  are three patterns that match exactly those sequences of characters (note that writing one character after another means "followed by" as usual).

• Characters (even special characters, except backslash followed by a double quote) enclosed in double quotes match themselves. For example, the following patterns are equivalent to the three given above:

         "abc"
         "=="
         "while"
         

  And the following pattern:

         "a|b"
         

  matches the three-character sequence: a then | then b, rather than matching a single a or a single b.

• The following characters have the usual special meanings as regular expression operators:

  |	means "or"
  *	means zero or more instances of
  +	means one or more instances of
  ?	means zero or one instance of
  ( )	are used for grouping

• The dot character matches any character except the newline character. It is usually used in the last rule in the specification, to match all "bad" characters (and the associated action issues an error message).

• The backslash is a special escape character:

  \n	newline
  \t	tab
  \"	double quote

  So to match a backslash character, put it in quotes. (See the JLex Reference Manual for a complete list of the special characters escaped by a backslash.)

• The up-arrow and dollar-sign characters: ^ and $, are special characters. When the up-arrow is used as the first character in a pattern, it causes the pattern to match only at the beginning of a line (i.e., only if the previous character was a newline). When the dollar sign is used as the last character in a pattern, it causes the pattern to match only at the end of a line (i.e., only if the next character is a newline).

• The regular expression can include character classes, delimited by square brackets:
  • A character class will match one character.
  • If no special characters are used inside the character class, then the character class matches any of the characters it includes inside its square brackets. For example: [abc] matches an a, or a b, or a c, so it is the same as: a|b|c.
  • Here are the characters that are "special" inside a character class:

    -	means a range of characters; e.g., a-z means "a to z".
    ^	is only a special character if it is the first character in the square brackets; it means not any of the following characters. So for example, [^abc] matches any character other than an a, or a b, or a c.
    \	is used as an escape character with n, b, etc as usual; it can also be used to escape the characters that are special inside a character class (e.g., [a\-z] matches an a or a - or a z, and [\\] matches a backslash.
    "	can be used around characters that are special inside a character class to make them match themselves (e.g., ["\"] matches a backslash, and ["-"] matches a hyphen.

Note that whitespace only matches itself if it is inside quotes or in a character class; otherwise, it ends the current pattern. So the two rules:

[a bc]    {}
a|" "|b|c {}

a bc   {}

Question 1: The character class [a-zA-Z] matches any letter. Write a character class that matches any letter or any digit.

Question 2: Write a pattern that matches any Pascal identifier (a sequence of one or more letters and/or digits, starting with a letter).

Question 3: Write a pattern that matches any Java identifier (a sequence of one or more letters and/or digits and/or underscores, starting with a letter or underscore).

Question 4: Write a pattern that matches any Java identifier that does not end with an underscore.

JLex directives

Recall that the second part of a JLex specification contains directives. This can include specifying the value that should be returned on end-of-file, specifying that line counting should be turned on, and specifying that the scanner will be used with the Java parser generator java cup. (See the JLex Reference Manual for more information about directives.)

The directives part also includes macro definitions. The form of a macro definition is:

name = regular-expression

DIGIT=		[0-9]
LETTER=		[a-zA-Z]
WHITESPACE=	[ \t\n]

Once a macro has been defined, it can be used in a regular expression (either to define another macro, or in the "Regular Expression Rules" part of the JLex specification. To use a macro, just use its name inside curly braces. For example, given the above macro definitions, the following pattern could be used to match Pascal identifiers:

{LETTER}({LETTER}|{DIGIT})*

Define a macro named NOTSPECIAL that matches any character except a newline, double quote, or backslash.

Comments

States

A quoted string consists of three parts:

1. A double quote.
2. Some text.
3. A double quote.

Use JLex states to write a specification to recognize quoted strings, and to return the number of escaped quotes in each such string.

yyline and yytext

A Small Example

%%

DIGIT=		[0-9]
LETTER=		[a-zA-Z]
WHITESPACE=	[ \t\n]      // space, tab, newline

// The next 3 lines are included so that we can use the generated scanner
// with java CUP (the Java parser generator)
%implements java_cup.runtime.Scanner
%function next_token
%type java_cup.runtime.Symbol

// Turn on line counting
%line

%%

{LETTER}({LETTER}|{DIGIT}*) {System.out.println(yyline+1 + ": ID " + yytext());}
{DIGIT}+                    {System.out.println(yyline+1 + ": INT");}
"="                         {System.out.println(yyline+1 + ": ASSIGN");}
"=="                        {System.out.println(yyline+1 + ": EQUALS");}
{WHITESPACE}*               { }
.               	    {System.out.println(yyline+1 + ": bad char");}

Quick Reference Guide