All Exams Test series for 1 year @ ₹349 only
Question

A lexical analyzer uses the following token definitions
  • $letter \rightarrow [A-Z a-z]$
  • $digit \rightarrow [0-9]$
  • $id \rightarrow letter(letter \mid digit)^*$
  • $number \rightarrow digit^+$
  • $ws \rightarrow (blank \mid tab \mid newline)^+$

For the string given below,

$x1 \quad 23mm \quad 78 \quad y \quad 7z \quad zz5 \quad 14A \quad 8H \quad AaYcD$

the number of tokens (excluding $ws$) that will be produced by the lexical analyzer is __________. (answer in integer)

The problem asks us to count the number of tokens produced by a lexical analyzer for a given string, based on specific token definitions. We need to exclude whitespace tokens ($ws$) from the final count.

Token Definitions Analysis

The lexical analyzer uses the following rules:

  • $letter$ $\rightarrow$ $[A-Z a-z]$ (Any uppercase or lowercase letter)
  • $digit$ $\rightarrow$ $[0-9]$ (Any digit from 0 to 9)
  • $id$ $\rightarrow$ $letter(letter | digit)*$ (Starts with a letter, followed by zero or more letters or digits)
  • $number$ $\rightarrow$ $digit+$ (One or more digits)
  • $ws$ $\rightarrow$ $(blank | tab | newline)+$ (One or more whitespace characters)

The lexical analyzer typically operates by scanning the input string from left to right and identifying the longest possible match for a token at each position.

String Tokenization Process

Let's trace the tokenization for the input string: $x1 23mm 78 y 7z zz5 14A 8H AaYcD$

  1. x1: Starts with 'x' (a $letter$). Followed by '1' (a $digit$). This matches the $id$ rule. Token 1 ($id$).
  2. (space): Matches $ws$. Ignored as per the question.
  3. 23mm:
    • Starts with '2' (a $digit$). The longest sequence of digits is '23'. Matches the $number$ rule. Token 2 ($number$).
    • Remaining string is 'mm'. Starts with 'm' (a $letter$). Followed by 'm' (a $letter$). Matches the $id$ rule. Token 3 ($id$).
  4. (space): Matches $ws$. Ignored.
  5. 78: Starts with '7' (a $digit$). The longest sequence of digits is '78'. Matches the $number$ rule. Token 4 ($number$).
  6. (space): Matches $ws$. Ignored.
  7. y: Starts with 'y' (a $letter$). Matches the $id$ rule (base case: letter followed by zero characters). Token 5 ($id$).
  8. (space): Matches $ws$. Ignored.
  9. 7z:
    • Starts with '7' (a $digit$). Matches the $number$ rule. Token 6 ($number$).
    • Remaining character is 'z'. Starts with 'z' (a $letter$). Matches the $id$ rule. Token 7 ($id$).
  10. (space): Matches $ws$. Ignored.
  11. zz5: Starts with 'z' (a $letter$). Followed by 'z' (a $letter$) and '5' (a $digit$). Matches the $id$ rule. Token 8 ($id$).
  12. (space): Matches $ws$. Ignored.
  13. 14A:
    • Starts with '1' (a $digit$). The longest sequence of digits is '14'. Matches the $number$ rule. Token 9 ($number$).
    • Remaining character is 'A'. Starts with 'A' (a $letter$). Matches the $id$ rule. Token 10 ($id$).
  14. (space): Matches $ws$. Ignored.
  15. 8H:
    • Starts with '8' (a $digit$). Matches the $number$ rule. Token 11 ($number$).
    • Remaining character is 'H'. Starts with 'H' (a $letter$). Matches the $id$ rule. Token 12 ($id$).
  16. (space): Matches $ws$. Ignored.
  17. AaYcD: Starts with 'A' (a $letter$). Followed by 'a', 'Y', 'c', 'D' (all $letters$). Matches the $id$ rule. Token 13 ($id$).

Final Token Count

Summing up the identified tokens (excluding $ws$): 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 = 13 tokens.

Was this answer helpful?

Important Questions from Lexical Analysis

  1. What is the output of a lexical analyzer?

  2. Which of the following are applications of symbol table ?

    (A) Storage allocation

    (B) Checking type compatability

    (C) Suppressing duplicate error messages

    Choose the correct answer from the options given below:

  3. Arrange the given compilation process in the correct order.

    a. Linking

    b. Assembling

    c. Compiling

    d. Pre-processing

  4. Which ONE of the following statements is FALSE regarding the symbol table?

Need Expert Advice?

Start Your Preparation with Prepp Mobile App

Download the app from Google Play & App Store
Download the app from Google Play & App Store
Prepp Mobile App