Newest questions tagged jflex - Stack Overflow

how to represent full Unicode range in regexp in JFlex?

2025-03-07T10:34:47Z

Is it possible in the current version of JFlex (1.9.1) to represent a range of full Unicode values in a regular expression ?

Something like this:

UnicodeIdentifier = [a-zA-Z_\u007F-\u10FFFF] [a-zA-Z0-9_\u007F-\u10FFFF]*

except this does not work (and makes JFlex emit a warning) because Unicode escape sequences in Java must be 16 bits in hexadecimal so the high end would be treated as \u10FF.

The spec says that representing supplementary characters in the range U+010000 to U+10FFFF requires two consecutive Unicode escapes however using this:

UnicodeIdentifier = [a-zA-Z_\u007F-\uDBFF\uDFFF] [a-zA-Z0-9_\u007F-\uDBFF\uDFFF]*

does not work either.

JFlex recognition conflict

2024-10-06T15:19:06Z

I'm developing a lexical analyzer and parser using JFlex and CUP. I’m running into a conflict in my lexer, and I’m having trouble understanding why it’s happening.

Here’s my lexer:

import java_cup.runtime.Symbol;
%%
%unicode
%cup
%line
%column

%eof{
    System.out.println("End of file");
%eof}

companyName = [A-Z][a-zA-Z0-9]*[0-9][a-zA-Z0-9]*
weekdays = Lundi|Mardi|Mercredi|Jeudi|Vendredi|Samedi|Dimanche|lundi|mardi|mercredi|jeudi|vendredi|samedi|dimanche
hours = [0-9]{1,2}h([0-9]{1,2})?
city = [A-Z][a-zA-Z -]+
number = [0-9]+

%%

Compagnie {
    System.out.println("Compagnie");
    return new Symbol(sym.COMPAGNIE);
}

{companyName} {
    System.out.println("Company name: " + yytext());
    return new Symbol(sym.COMPANY_NAME, yytext());
}

{weekdays} {
    System.out.println("Weekday: " + yytext());
    return new Symbol(sym.WEEKDAY, yytext());
}

"au depart de" {
    System.out.println("Departure");
    return new Symbol(sym.DEPART);
}

pour {
    System.out.println("For");
    return new Symbol(sym.FOR);
}

par {
    System.out.println("By");
    return new Symbol(sym.BY);
}

Fin {
    System.out.println("End");
    return new Symbol(sym.FIN);
}

==== {
    System.out.println("Separator");
    return new Symbol(sym.SEPARATOR);
}

: {
    System.out.println("Colon");
    return new Symbol(sym.COLON);
}

car {
    System.out.println("Car");
    return new Symbol(sym.CAR);
}

= {
    System.out.println("Equal");
    return new Symbol(sym.EQUAL);
}

, {
    System.out.println("Comma");
    return new Symbol(sym.COMMA);
}

"(" {
    System.out.println("Open parenthesis");
    return new Symbol(sym.OPEN_PARENTHESIS);
}

")" {
    System.out.println("Close parenthesis");
    return new Symbol(sym.CLOSE_PARENTHESIS);
}

{hours} {
    System.out.println("Hours: " + yytext());
    return new Symbol(sym.HOURS, yytext());
}

{number} {
    System.out.println("Number: " + yytext());
    return new Symbol(sym.NUMBER, Integer.parseInt(yytext()));
}

{city} {
    System.out.println("City: " + yytext());
    return new Symbol(sym.CITY, yytext());
}

[\n\t\r\s]+ {}  // Skip whitespace

. {
    System.out.println("Error: " + yytext() + " at line " + yyline + ", column " + yycolumn);
}

Here’s the input I’m testing:

Compagnie Bloblo007 au depart de Brest ====
Mardi :
    8h = car 2733 pour Nantes (par Quimper),
    15h10 = car 902 pour Rennes (par Morlaix, Saint-Brieuc, Montauban-de-Bretagne),
    09h00 = car 1203 pour Saint-Malo

lundi :
    12h = car 80862 pour Landerneau,
    8h5 = car 70 pour Bordeaux (par Vannes, La Roche-Bernard, Nantes,
    La Roche-sur-Yon, Niort),
    15h15 = car 82019 pour Paris (par Quimper, Nantes)

Fin

The issue arises on the first line. Specifically, Compagnie Bloblo007 is not being correctly matched as COMPAGNIE and COMPANY_NAME. Instead, Compagnie Bloblo is being recognized as a CITY and 007 as a NUMBER. However, if I remove the city and number rules, Compagnie and companyName match correctly.

Question:

How can I adjust my lexer to correctly match the Compagnie keyword and the entire company name (Bloblo007) without mistakenly treating the text as a city or number?

Thanks in advance for your help!

Syntax fragment (include or import) in YACC

2024-04-07T20:18:26Z

Is it possible to include/import yacc fragment files from different files to a main YACC?

Just to exemplify what I'm looking for, I would like to create 3 syntax parsers for 3 different files, but they'd share a common syntax block.

So, I'd like to keep this syntax in only one yacc fragment file, so I could maintain it better.

JFlex RE matching lookahead and precedence

2023-09-12T16:45:07Z

For XPath parsing, I need to distinguish the token type of "name" in the cases where it is followed by "::" (possibly after whitespace), followed by "(" (possibly after whitespace), or followed by neither.

In JLex we did this with a routine that read ahead in the yy_* buffer, but that isn't exposed in JFlex, and a lookahead RE should be a cleaner solution than handcoded lookahead.

Unfortunately, my first attempt isn't working as expected; it's falling into the third category (standalone name) more often than I would have expected.

What I'm trying is the following pattern, where "self" is the thing I'm trying to separate between the two cases. Please pardon the "GONK_" -- that's my convention for debugging wrappers. And the wrappered newSymbol() is itself a wrapper for new Symbol() which has a side effect; sorry about that.

"self/\s*::"       { return GONK_newSymbol(sym.SELF); }
"self/\s*[(]"      { return GONK_newSymbol(sym.SELF); }
"self"             { return GONK_newSymbol(sym.QNAME,yytext()); }

As I understand it, JFlex's RE rules are "longest match wins, ties broken in favor of first match", so I expected the lookaheads to take precedence. But putting a trace printout in the GONK_ functions tells me that the third case (sym.QNAME) is almost always being taken.

I'm sure my error is obvious and I'm looking right at it, but... What did I miss?

How to set a package name for generated classes when using cup.gradle.cup-gradle-plugin?

2023-06-09T02:04:42Z

The cup arg: -destdir does not work. With or without this option, the generated source files (parser and symbols) always in the top folder (build/generated-src/cup).

For example as below, the expected destination folder is 'build/generated-src/cup/my/pkg' but actually it's still 'build/generated-src/cup'.

cup {
   args = ["-destdir", "build/generated-src/cup/my/pkg", "-parser", "my_parser", "-symbols", "my_sym"]
}

The questions are:

How can I get the generated class in the right folder that matching the package name?
Is there a workaround to move the generated files by using a gradle task?

How can I use JFlex if i´m cloning a project from github [closed]

2023-02-08T19:57:48Z

i'm currently working on a Lexer in Java, i'm using jflex to do this, when my partner runs the code it works, he pushed it into a repo in github. I cloned it and when trying to run it, it says it cannot find a symbol which is Lexer, this is a class that jflex creates. I

I was expecting to be able to use the jflex library, which is stored in the lib directory in the project, also, in the .vscode directory we created a settings.json in order to be able to use it, we thought it worked but apparently when someone clones the repo it doesn't work. How can i solve this?

JetBrains Language Support Plugin

2022-07-28T16:08:57Z

I am trying to implement language support plugin for a basic language. I am following jetbrains's tutorial for simple language support (.properties file support basically) and on the side I have rust plugin for reference. However the complexity gap between them is huge so some questions are hard to find answers to from both sources.

Here is my question: What is the best way to allow spaces between tokens which DO NOT require spaces, however force spacing between tokens which do?

i.e. class Foo{ <- here first space is mandatory, but the second one (before '{' symbol) can be ommited.

JFLEX regular expressions for string starts with

2022-06-22T07:39:47Z

I'm new with this jflex and regular expressions I read a lot even though I know I faced the solution in the manual but really I can't understand these regular expressions. I want simple read a text file with jflex and return every line that starts with name = or name= but this was a pain for me for 1 week I'm reading all regular expressions in the jflex document but I can't understand which one I can use to achieve it anyone can help me out? with jflex rules and any code to contain that?

this is the example input and output

bla bla bla
bla bla bla
name=StackOverFlow
name = ThisIsGo
bla bla bla

Output:

StackOverFLow

ThisIsGo

Unexpected character JFLEX

2022-06-21T15:14:32Z

Hello i was watching a tutorial on youtube. It was the start of tutorial but when i tried to wrote the code and run this flex.l file. The output show unexpected character [a-z] {printf("Single lowercase character\n");} . What should i do with this :(

the youtube link https://www.youtube.com/watch?v=LpVufkH4gog&t=171s time stamp 2:54

   %{
    
     /* definitions*/
    
    %}
    
    /*rules*/
    
    %%
    
    [a-z] {printf("Single lowercase character\n");}
    . {printf("Not a lowercase character\n");}
    \n {return 0;}
    
    %%
    
    
    yywrap() {}
    
    int main() {
    
        printf("Enter String:  ");
        yylex();
        
        return 0;
    
    }

Regular expression for email in flex

2022-06-01T17:54:27Z

I am trying to wirte a regular expression for emails in JFlex. So far I tried with this

L=[a-zA-Z_]+
D=[0-9]+    
email=[^(.+)@(\S+)$]
%{
    public String lexeme;
%}
%%
{L}({L}|{D})* {lexeme=yytext(); return Identi;}
("(-"{D}+")")|{D}+ {lexeme=yytext(); return Number;}
{email} {lexeme=yytext(); return Email;}
 . {return ERROR;}

When I test with an email, the lexer is not matching any email. How to match email?

Jflex ambiguity

2022-05-08T19:58:20Z

I have these two rules from a jflex code:

Bool = true
Ident = [:letter:][:letterdigit:]*

if I try for example to analyse the word "trueStat", it gets recognnized as an Ident expression and not Bool. How can I avoid this type of ambiguity in Jflex?

Why does my $1 operation not work with Java Bison?

2022-04-16T15:23:01Z

When I have an abstract syntax tree in the Java Bison (.y) file with this:

string_op:
  SYMBOL_PLUS { $$ = $1; System.out.println("this should print + here: " + $$);}
  | SYMBOL_DOUBLEEQ
  | SYMBOL_NOTEQ;

I would expect it to print "this should print + here:+" whenever the parser comes across the + token (i.e. SYMBOL_PLUS). However, I always get "this should print + here:null".

I have already used the %type declarations and everything.

%nterm <String> string_op

%type<String> SYMBOL_PLUS
%type<String> RESERVED_BOOL
%type<String> RESERVED_STRING
%type<String> RESERVED_INT
%type<String> IDENTIFIER
%type<String> SYMBOL_DOUBLEEQ
%type<String> SYMBOL_NOTEQ

It still gives the error. There is a massive lack of good, understandable, and clear documentation for java bison so it's really frustrating. Please let me know if you can fix this issue. I've been on it for days.

JFlex complaining about caret character

2022-03-30T16:31:29Z

I'm trying to produce a simple standalone scanner with the use of JFlex. What I want to do now is to simply recognize the beginning of a line with the use of the caret (^) symbol.

From the Semantics section of the user manual:

In a lexical rule, a regular expression r may be preceded by a ^ (the beginning of line operator).

This is my code right now:

// lexer.jflex

%%
%standalone
%class Lexer

beginning = ^

%%

{beginning}     { System.out.println("beginning"); }
.                 { ; }

The required behavior is very simple: it should only recognize the beginning of each input line and ignore everything else. However, for some reason, jflex complains about the caret symbol:

Error in file "lexer.jflex" (line 5):
Syntax error.
beginning = ^
            ^

I also tried with some variations, for example with double quotes around the caret ("^") and escape character before it (\^), but obviously this recognizes the caret symbol as a character itself, and not as a regular expression representing the beggining of a line.

EDIT

Problem solved: the caret symbol must not be used in the options and macros section but instead in the actions and rules section, i.e. the third one. For example, the following code matches the # symbol at the beginning of a line:

// lexer.jflex

%% 
...
hash = "#"

%%

^{hash} { System.out.println("hash found!"); }

Java Bison and Jflex error for redeclared/undeclared variables

2022-03-19T12:38:14Z

I am making a compiler with Jflex and Bison. Jflex does the lexical analysis. Bison does the parsing.

The lexical analysis (in a .l file) is perfect. Tokenizes the input, and passes the input to the .y file for Bison to parse.

I need the parser to print an error for redeclared/undeclared variables. My thought are that it would need some sort of memory to remember all the variables initialized so far, so that it can produce an error for those tokens coming in and when it sees an undeclared variable being used. For example, ''bool", "test", "=", "true", ";", and on a new line, "test2", "=", "false", ";", the parser would need some sort of memory to remember ''test" and when it parses the second line it can access that memory again and say that "test2" is undeclared, hence it would print an error.

What I'm confused about is how we can make a memory like that with bison using Java in the .y file. With C, you would use the -d flag and it would make 2 files with enum types and a header file which would keep track of the declared variables but in Java I'm not too sure if I can do the same as I can't structure the grammar in any way so that it will remember variable names.

I could make a symbol table in Java code to check for redeclared variables, but in the main() in the .y file I have

 public static void main(String args[]) throws IOException {
    EXAMPLELexer lexer = new EXAMPLLexer(System.in);
    EXAMPLE parser = new EXAMPLE(lexer);
    if(parser.parse()){
      System.out.println("VALID FROM PARSER");
    }
    else{
      System.out.println("ERROR FROM PARSER");
    }
      
    return;
  }

There is no way to get the tokens individually and pass them into another java instance or whatever.%union{} doesnt work with Java, so I dont know how this is even possible. I can't find a single piece of documentation explaining this so I would love some answers!

How to make expression for grammar rule in Bison with Jflex

2022-03-17T20:24:16Z

I've been trying to make a parser which takes tokens from a lexer (jflex), and ive used Java with bison for the parser. Here is my code so far for the parser:

%define api.prefix {EXAMPLE}
%define api.parser.class {EXAMPLE}
%define api.parser.public
%define parse.error verbose

%code imports{
  import java.io.InputStream;
  import java.io.InputStreamReader;
  import java.io.Reader;
  import java.io.IOException;
}

%code {
  public static void main(String args[]) throws IOException {
    EXAMPLELexer lexer = new EXAMPLELexer(System.in);
    EXAMPLE parser = new EXAMPLE(lexer);
    if(parser.parse()){
      System.out.println("VALID FROM PARSER");
    }
    else{
      System.out.println("ERROR FROM PARSER");
    }
      
    return;
  }
}

/* Bison Declarations */
%token <Integer> NUM
%type exp
%%

input: line | input line;
line: '\n'
| exp '\n' { System.out.println($exp); }
| error '\n'
;
exp:
 NUM { $$ = $1; }
| exp '=' exp { if ($1.intValue() != $3.intValue()) yyerror("calc: error: " + $1 + " != " + $3); }
| exp '+' exp { $$ = $1 + $3; }
| exp '-' exp { $$ = $1 - $3; }
| exp '*' exp { $$ = $1 * $3; }
| exp '/' exp { $$ = $1 / $3; }
| '-' exp %prec NEG { $$ = -$2; }
| exp '^' exp { $$ = (int) Math.pow($1, $3); }
| '(' exp ')' { $$ = $2; }
| '(' error ')' { $$ = 1111; }
| '!' { $$ = 0; return YYERROR; }
| '-' error { $$ = 0; return YYERROR; }

%%

class EXAMPLELexer implements EXAMPLE.Lexer {
  InputStreamReader it;
  Yylex yylex;

  public EXAMPLELexer(InputStream is){
    it = new InputStreamReader(is);
    yylex = new Yylex(it);
  }

  @Override
  public void yyerror (String s){
   
  }

  @Override
  public Object getLVal() {
    return null;
  }

  @Override
  public int yylex () throws IOException{
    return yylex.yylex();
  }
}

When I try to use the jflex and bison commands I get this: $$ of ‘exp’ has no declared type, same with $1, $3

Please help, i've been stuck on this for a while! (p.s i've already tried %union{} and it just gave rise to more errors unfortunately, if you have any sample code that actually works with it, i'd love to learn as there seems to be a lack of documentation with Java and bison working with Jflex.)

How to use Jflex and Bison Together?

2022-03-16T22:47:26Z

I am really struggling to use Jflex and Bison together. For example, here's some sample code:

https://github.com/valecor95/bison-flex-jflex-examples/tree/master/Java/1_BalancedParentheses/BalPar1

You run make:

It just makes a load of files, and then if I want to test to see if it succeeds with the parser (stuff in a file called test.txt), I just can't. It's forced to take in System.in. It's really frustrating as there's very little readable and simple documentation for JFLEX and bison.

Any advice at all would be much appreciated.

Parsing grammar error with JCup and JFlex when run parser

2022-02-21T09:40:06Z

Continuing with the example that I am doing with JCup and JFlex Error when run parsing using cup and jflex, I am getting an error when parsing the following statement.

Label(name="Label1", title="Label title");

I don't understand the reason for the error, but I have looked for examples of how it does it in java15.cup. I understand that it is not the same, but it seems to be an approximation. When the Lexer is executed, when showing the tokens found by console, it does not print the COMMA token, I do not understand the reason if it has the same SEMMICOLON configuration. I think there is something in the grammar that I don't understand and so I ask for help to see if I can solve it or find another solution.

Macro declaration on jflex file

LineTerminator = \r|\n|\r\n
InputCharacter = [^\r\n]
WhiteSpace     = {LineTerminator}+ | [ ,\t,\f]+

/* comments */
Comment = {TraditionalComment} | {EndOfLineComment} | {DocumentationComment}

TraditionalComment   = "/*" [^*] ~"*/" | "/*" "*"+ "/"
// Comment can be the last line of the file, without line terminator.
EndOfLineComment     = "//" {InputCharacter}* {LineTerminator}?
DocumentationComment = "/**" {CommentContent} "*"+ "/"
CommentContent       = ( [^*] | \*+ [^/*] )*

/* General identifiers */
Identifier          = [A-Za-z_$] [A-Za-z_$0-9]*

StringConst         = \" ([^\"])* \"

/*
    identifiers
*/
ComponentId     = \" {Identifier} \"

/*
    string to assign values, title, etc
*/
StringLit           = {StringConst}

Some Terminal

"["             { return iSymbol(Token.LBRCKT.name(), Symbol.LBRCKT, yytext()); }
"]"             { return iSymbol(Token.RBRCKT.name(), Symbol.RBRCKT, yytext()); }
";"             { return iSymbol(Token.SEMICOLON.name(), Symbol.SEMICOLON, yytext()); }
","             { return iSymbol(Token.COMMA.name(), Symbol.COMMA, yytext()); }
"."             { return iSymbol(Token.DOT.name(), Symbol.DOT, yytext()); }
"="             { return iSymbol(Token.ASSIGN.name(), Symbol.ASSIGN, yytext()); }
":"             { return iSymbol(Token.COLON.name(), Symbol.COLON, yytext()); }
"("             { return iSymbol(Token.LPAREN.name(), Symbol.LPAREN, yytext()); }
")"             { return iSymbol(Token.RPAREN.name(), Symbol.RPAREN, yytext()); }
"{"             { return iSymbol(Token.LCBRACE.name(), Symbol.LCBRACE, yytext()); }
"}"             { return iSymbol(Token.RCBRACE.name(), Symbol.RCBRACE, yytext()); }

The grammar defined for the statement is as follows;

name_property_def   ::= NAME ASSIGN STRING_ID ;

title_property_def  ::= TITLE ASSIGN STRING_LIT ;


label_basic_declaration ::=
        IQ_LABEL LPAREN label_formal_parameter_list_opt RPAREN SEMICOLON 
    ;

label_formal_parameter_list_opt ::=
    |   label_formal_parameter_list
    ;
    
label_formal_parameter_list ::=
        label_formal_parameter_list COMMA label_formal_parameter
    |   label_formal_parameter
    ;
    
label_formal_parameter ::=
        name_property_def
    |   title_property_def
    ;

the values that go between parentheses are fixed, that is, they are the names of the "properties" that this object has. I appreciate the help

Lexer trace 
Token: Symbol: LABEL (unknown:11/2(-1) - unknown:11/8(-1)) Label 
Token: Symbol: LPAREN (unknown:11/9(-1) - unknown:11/9(-1)) (
Token: Symbol: NAME (unknown:11/10(-1) - unknown:11/13(-1)) name
Token: Symbol: ASSIGN (unknown:11/14(-1) - unknown:11/14(-1)) = 
Token: Symbol: STRING_ID (unknown:11/15(-1) - unknown:11/24(-1)) "Label1"
Token: Symbol: TITLE (unknown:11/26(-1) - unknown:11/30(-1)) title
Token: Symbol: ASSIGN (unknown:11/31(-1) - unknown:11/31(-1)) = 
Token: Symbol: STRING_LIT (unknown:11/32(-1) - unknown:11/46(-1)) "Label title"
Token: Symbol: RPAREN (unknown:11/47(-1) - unknown:11/47(-1)) )
Token: Symbol: SEMICOLON (unknown:11/48(-1) - unknown:11/48(-1)) ;

# Initializing parser
# Current Symbol is #19
# Shift under term #19 to state #2
# Current token is Symbol: LPAREN (unknown:2/8(-1) - unknown:2/8(-1))
# Shift under term #3 to state #5
# Current token is Symbol: NAME (unknown:2/9(-1) - unknown:2/12(-1))
# Shift under term #33 to state #21
# Current token is Symbol: ASSIGN (unknown:2/13(-1) - unknown:2/13(-1))
# Shift under term #11 to state #25
# Current token is Symbol: STRING_ID (unknown:2/14(-1) - unknown:2/23(-1))
# Shift under term #15 to state #26
# Current token is Symbol: RPAREN (unknown:2/24(-1) - unknown:2/24(-1))
# Reduce with prod #10 [NT=6, SZ=3]
# Reduce rule: top state 5, lhs sym 6 -> state 40
# Goto state #40
# Reduce with prod #6 [NT=4, SZ=1]
# Reduce rule: top state 5, lhs sym 4 -> state 39
# Goto state #39
# Shift under term #4 to state #41
# Current token is Symbol: LCBRACE (unknown:2/26(-1) - unknown:2/26(-1))
# Reduce with prod #4 [NT=5, SZ=3]
# Reduce rule: top state 2, lhs sym 5 -> state 6
# Goto state #6
# Shift under term #5 to state #7
# Current token is Symbol: LABEL (unknown:11/2(-1) - unknown:11/8(-1))
# Shift under term #21 to state #9
# Current token is Symbol: LPAREN (unknown:11/9(-1) - unknown:11/9(-1))
# Shift under term #3 to state #15
# Current token is Symbol: NAME (unknown:11/10(-1) - unknown:11/13(-1))
# Shift under term #33 to state #21
# Current token is Symbol: ASSIGN (unknown:11/14(-1) - unknown:11/14(-1))
# Shift under term #11 to state #25
# Current token is Symbol: STRING_ID (unknown:11/15(-1) - unknown:11/24(-1))
# Shift under term #15 to state #26
# Current token is Symbol: TITLE (unknown:11/27(-1) - unknown:11/31(-1))
Compiler has detected a syntax error at line 0 column 0
# Attempting error recovery
# Finding recovery state on stack
# Pop stack by one, state was # 26
# Pop stack by one, state was # 25
# Pop stack by one, state was # 21
# Pop stack by one, state was # 15
# Pop stack by one, state was # 9
# Pop stack by one, state was # 7
# Pop stack by one, state was # 6
# Pop stack by one, state was # 2
# Pop stack by one, state was # 0
# No recovery state found on stack
# Error recovery fails
Couldn't repair and continue parse for input symbol "TITLE" spanning from unknown:11/27(-1) to unknown:11/31(-1)

It could be that the StringLit or ComponentId macro is causing the problem.

Jflex and CUP not working on vscode and mac

2022-02-18T21:24:58Z

I downloaded jflex by doing brew install jflex everything worked the way it should.

But now when I'm trying to make it work with java_cup.runtime.*. And I keep getting the errors below

Lexer.java:681: error: cannot find symbol
          { return new java_cup.runtime.Symbol(sym.EOF); }
                                               ^
  symbol:   variable sym
  location: class Lexer

My proffesor said,

"The Symbol class is part of the parser (JavaCUP)’s runtime jar file. You may use it in the lexer simply by setting the classpath to include the jar file from JavaCUP and importing it"

So I made bash file below and did what he said and it is not working.

#!/bin/bash
jflex MiniJava.jflex
javac -cp "/Users/carlosfield/Desktop/School/csc453/java-cup-bin-11b-20160615/java-cup-11b.jar" Lexer.java
java -cp "/Users/carlosfield/Desktop/School/csc453/java-cup-bin-11b-20160615/java-cup-11b-runtime.jar"  Lexer Factorial.java
rm Lexer*

This is my jflex file

import java.util.*;
import java_cup.runtime.*;
%% 
%class Lexer
%cup
%line
%column
%{
private Symbol symbol(int type) {
    return new Symbol(type, yyline, yycolumn);
}
private Symbol symbol(int type, Object value) { 
    return new Symbol(type, yyline, yycolumn, value); 
} 
%} 
WhiteSpace = [ \t\r\n]
Identifier = [a-zA-Z_][a-zA-Z0-9_]*
Integer = 0 | [1-9][0-9]* 
%% 
“+” {  }

Jflex Regular expression for string with escaped characters

2022-02-16T23:53:30Z

I am generating lexical analyzer with JFLEX. I need to find all strings which containing escaped characters using regular expressions. I'm struggling with this. What could it be?

Error when run parsing using cup and jflex

2022-01-07T21:00:45Z

I am new to JFlex and CUP. I am trying to do a simple example, but when I run the parser, it always gives the same error, it does not progress with the recognition of the statements. I think the problem must be in the productions or rules. I have defined as terminals the symbols that are always used in Java, for example:

terminal LPAREN, RPAREN, RBRACE, LBRACE, LBRCKT, RBRCKT, COLON, SEMICOLON, ASSIGN, COMMA, DOT;
terminal PAGE, LABEL;

I define non-terminals as follows:

nonterminal START_PAGE;
nonterminal page_body, page_body_declarations_opt, page_body_declarations, page_body_declaration;

The grammar like this:

start with START_PAGE;
START_PAGE ::= PAGE page_body ;

page_body ::= LBRACE page_body_declarations_opt RBRACE ;
page_body_declarations_opt ::= | page_body_declarations ;
page_body_declaration ;
page_body_declarations ::= page_body_declarations page_body_declaration ;
page_body_declaration ::= label_declaration ;
label_declaration ::= LABEL LPAREN RPAREN SEMICOLON ;

The input data or file contains the following from line 2:


Page {
}

When I run the parser, I print the result of the lexer, then I run the parser getting the following result:

Token: # 2 Page
Token: # 51 {
Token: # 52}
Token: # 0
Compiler has detected a syntax error at line 2 column 8
Error in line 2, column 8: Couldn't repair and continue parse

The error is presented in the LBRACE.

The version I am using for testing is:

<jflex.version>1.8.2</jflex.version>
<cup.version>11b-20160615</cup.version>

The general idea is:

Define the page block and internally define other elements such as the label.

I would be grateful if you would give me a hand to be able to carry out the example.

The information that exists is very scarce for the definition of rules or productions and how the reductions in CUP should be made. All the examples that I have investigated are arithmetic expressions but I have not found more that can contribute to the resolution of the problem.

This is my lexer file


// USER CODE

package core;

import java.io.Reader;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java_cup.runtime.Symbol;
import java_cup.runtime.Scanner;
import java.nio.charset.StandardCharsets;
import java_cup.runtime.ComplexSymbolFactory;
import java_cup.runtime.ComplexSymbolFactory.Location;
import java_cup.runtime.ComplexSymbolFactory.ComplexSymbol;


/**
 * Lexer class.
 */

%% /*----------------------------------------------------------*/

%public
// Lexer class to generate
%class Lexer
%cupsym MSymbol
%function next_token
%implements MSymbol, Scanner
%type java_cup.runtime.Symbol

%unicode

%cupdebug

%char
%full

%line
%column

%eofval{
    
    return mSymbol(MSymbol.EOF);

%eofval}


/*--------------------------------------------------------------
    CODE COPIED INTO LEXER
  --------------------------------------------------------------*/
%{ 

    ComplexSymbolFactory symbolFactory;
    
    StringBuffer string = new StringBuffer();

    public Lexer(Reader in, ComplexSymbolFactory sf){
        this(in);
        symbolFactory = sf;
    }
    
    private Symbol mSymbol(int type) {
        return new Symbol(type, yyline, yycolumn);
    }
    
    private Symbol mSymbol(int type, Object value) {
        return new Symbol(type, yyline, yycolumn, value);
    }
    private void error(String message) {
        System.out.println("Error at line "+(yyline+1)+", column "+(yycolumn+1)+" : " + message);
    }

%}

/*--------------------------------------------------------------
    MACRO DECLARATIONS
  --------------------------------------------------------------*/
LineTerminator = \r|\n|\r\n
InputCharacter = [^\r\n]
//WhiteSpace     = {LineTerminator} | [\ ,\t,\f]
WhiteSpace     = [\ ,\t,\f,\t] | {LineTerminator}

/* comments */
Comment = {TraditionalComment} | {EndOfLineComment} | {DocumentationComment}

TraditionalComment   = "/*" [^*] ~"*/" | "/*" "*"+ "/"
// Comment can be the last line of the file, without line terminator.
EndOfLineComment     = "//" {InputCharacter}* {LineTerminator}?
DocumentationComment = "/**" {CommentContent} "*"+ "/"
CommentContent       = ( [^*] | \*+ [^/*] )*


%state STRING

%% /*----------------------------------------------------------*/

/* Keywords */

<YYINITIAL> {

/*-------------------------------------------------------------
    KEYWORDS
  -------------------------------------------------------------*/
    "Page"      { 
                    return mSymbol(MSymbol.PAGE, yytext()); 
                }
    
}   //------> End of Keywords


<YYINITIAL> {
    /* separators */
    "("             { return mSymbol(MSymbol.LPAREN, yytext()); }
    ")"             { return mSymbol(MSymbol.RPAREN, yytext()); }
    "{"             { return mSymbol(MSymbol.RBRACE, yytext()); }
    "}"             { return mSymbol(MSymbol.LBRACE, yytext()); }
    "["             { return mSymbol(MSymbol.LBRCKT, yytext()); }
    "]"             { return mSymbol(MSymbol.RBRCKT, yytext()); }
    ";"             { return mSymbol(MSymbol.SEMICOLON, yytext()); }
    ","             { return mSymbol(MSymbol.COMMA, yytext()); }
    "."             { return mSymbol(MSymbol.DOT, yytext()); }
    "="             { return mSymbol(MSymbol.ASSIGN, yytext()); }
    ":"             { return mSymbol(MSymbol.COLON, yytext()); }

    \"              { yybegin(STRING); string.setLength(0); }
    

    /* WHITESPACE */
    {WhiteSpace}    { /* ignore */ }

    /* comments */
    {Comment}       { /* ignore */ }
}

<STRING> {
    \"              {   
                        yybegin(YYINITIAL);
                        return mSymbol(MSymbol.STRING_LITERAL, string.toString()); 
                    }
    
}

/* error fallback */
[^]                 { 
                        this.error("Illegal character [ " + yytext() + " ]");
                    }

When run the parser, the lexer detect tokens. I don't know what happens with space.

Token: #2 Page
Token: #51 {
Token: #52 }
Token: #0 
Compiler has detected a syntax error at line 2 column 8
Error in line 2, column 8 : Couldn't repair and continue parse

The text Couldn't repair and continue parse is set by parser cup.

This is the result when generating parser

------- CUP v0.11b 20160615 (GIT 4ac7450) Parser Generation Summary -------
  0 errors and 54 warnings
  62 terminals, 7 non-terminals, and 8 productions declared, 
  producing 15 unique parse states.
  54 terminals declared but not used.
  0 non-terminals declared but not used.
  0 productions never reduced.
  0 conflicts detected (0 expected).
  Code written to "Parser.java", and "MSymbol.java".
---------------------------------------------------- (CUP v0.11b 20160615 (GIT 4ac7450))

Code to run parser

    ComplexSymbolFactory csf = new ComplexSymbolFactory();
    Lexer lexer = new Lexer(new BufferedReader(new FileReader(args[0], StandardCharsets.UTF_8)), csf);
    ScannerBuffer lxrBuff = new ScannerBuffer(lexer);
    Parser mParser = new Parser(lxrBuff, csf);
    mParser.parse();

Thanks in advance

Regular expressions representing BNF

2021-11-27T17:33:39Z

I am working on a lexer. I need to identify patterns in BNF and specify them using Regular expressions. I know there are TOKENS e.g. keywords, identifiers, operators etc. So far I have defined Regular expressions:

 digit=[0-9]
    integer={digit}+
    letter=[a-zA-Z]

But the given BNF rule for Identifier is:

   < id > ::= < letter >
                | "_"
                | < id > < digit >
                | < id > < letter >

Since the <id> is defined by Recursive pattern, how can I express this pattern using Regular expression. I tried this Regular expression: id={letter}+|"_"|{id}{digit}+ for the above BNF rule but it is giving me error that Regular expression contains cycle.

check if condition is met before executing the action in JFlex

2021-10-17T09:39:06Z

I am writing a lexical analyzer using JFlex. When the word co is matched, we have to ignore what comes after until the end of the line (because it's a comment). For the moment, I have a boolean variable that changes to true whenever this word is matched and if an identifier or an operator is matched after co until the end of the line, I simply ignore it because I have an if condition in my Identifier and Operator token identification.
I am wondering if there is better way to do this and get rid of this if statement that appears everywhere?

Here is the code:

%% // Options of the scanner

%class Lexer     
%unicode        
%line      
%column      
%standalone 

%{
    private boolean isCommentOpen = false;
    private void toggleIsCommentOpen() {
        this.isCommentOpen = ! this.isCommentOpen;
    }
    private boolean getIsCommentOpen() {
        return this.isCommentOpen;
    }
%} 

Operators           = [\+\-]
Identifier          = [A-Z]*

EndOfLine           = \r|\n|\r\n

%%
{Operators}         {
                        if (! getIsBlockCommentOpen() && ! getIsCommentOpen()) {
                            // Do Code
                        }
                    }  
 
{Identifier}        {
                        if (! getIsBlockCommentOpen() && ! getIsCommentOpen()) {
                            // Do Code
                        }
                    }

"co"                {
                        toggleIsCommentOpen();
                    }

.                   {}

{EndOfLine}         {
                        if (getIsCommentOpen()) {
                            toggleIsCommentOpen();
                        }
                    }

JFlex - lex structured document

2021-08-28T09:50:26Z

The following image represents code that I need to lex.
The document has the following format (in columns term):

 1 -  5:     comment
 6 - 79:     actual code
80 - ..:     comment

If I only had to lex the middle part, there would be no issues at all.
Unfortunately, the initial and terminating comment are always present in the document.

Any ideas on how this could be implemented?
I was thinking about implementing a two-phases lexer, but my thoughts are a bit confused still.

Highlight / parse PSI elements inside another PSI element

2021-07-28T12:58:51Z

I developed a Custom Language plugin based on this this tutorial.

My plugin parses key/value language files with format like below. Values can contain some HTML tags like <br>, <i>, <b>, <span> and \n. So I want to highlight these tags as separate PSI elements inside green PSI elements (values) (see pic). How can I overwrite my rules to get this?

#Section header

KEY1 = First<br>Value
KEY2 = Second\nValue

Bnf rules I use

lngFile ::= item_*

private item_ ::= (property|header|COMMENT|CRLF)

property ::= (KEY? SEPARATOR VALUE?) | KEY {
  mixin="someClass"
  implements="someClass"
  methods=[getKey getValue getName setName getNameIdentifier getPresentation]
}

header ::= HEADER {
  mixin="someClass"
  implements="someClass"
  methods=[getName setName getNameIdentifier getPresentation]
}

Flex

%%

%class LngLexer
%implements FlexLexer
%unicode
%function advance
%type IElementType
%eof{  return;
%eof}

CRLF=\R
WHITE_SPACE=[\ \n\t\f]
FIRST_VALUE_CHARACTER=[^ \n\f\\] | "\\"{CRLF} | "\\".
VALUE_CHARACTER=[^\n\f\\] | "\\"{CRLF} | "\\".
END_OF_LINE_COMMENT=("//")[^\r\n]*
HEADER=("#")[^\r\n]*
SEPARATOR=[:=]
KEY_CHARACTER=[^:=\ \n\t\f\\] | "\\ "

%state WAITING_VALUE

%%

<YYINITIAL> {END_OF_LINE_COMMENT}                           { yybegin(YYINITIAL); return LngTypes.COMMENT; }

<YYINITIAL> {HEADER}                                        { yybegin(YYINITIAL); return LngTypes.HEADER; }

<YYINITIAL> {KEY_CHARACTER}+                                { yybegin(YYINITIAL); return LngTypes.KEY; }

<YYINITIAL> {SEPARATOR}                                     { yybegin(WAITING_VALUE); return LngTypes.SEPARATOR; }

<WAITING_VALUE> {CRLF}({CRLF}|{WHITE_SPACE})+               { yybegin(YYINITIAL); return TokenType.WHITE_SPACE; }

<WAITING_VALUE> {WHITE_SPACE}+                              { yybegin(WAITING_VALUE); return TokenType.WHITE_SPACE; }

<WAITING_VALUE> {FIRST_VALUE_CHARACTER}{VALUE_CHARACTER}*   { yybegin(YYINITIAL); return LngTypes.VALUE; }

({CRLF}|{WHITE_SPACE})+                                     { yybegin(YYINITIAL); return TokenType.WHITE_SPACE; }

[^]                                                         { return TokenType.BAD_CHARACTER; }

Explanation of JFlex Block Comment rule

2021-05-23T15:48:49Z

I was looking on how to implement block comments in JFlex for custom language support in intellij and found that it can be described as

"/*" !([^]* "*/" [^]*) ("*/")?

I don't quite understand how to read this and would like it if it were explained in plain English.

At the moment I'm reading this as

first expect a /* then
expect not
- any character? (Not sure why they used [^]) zero or more times
- followed */
- Any character zero or more
An optional */

JFlex maximum read length

2021-05-16T16:15:55Z

Given a positional language like the old IBM RPG, we can have a line such as

CCCCCDIDENTIFIER     E S             10

Where characters

 1-5:  comment
   6:  specification type
7-21:  identifier name
...And so on

Now, given that JFlex is based on RegExp, we would have a RegExp such as:

[a-zA-Z][a-zA-Z0-9]{0,14} {0,14}

for the identifier name token.
This RegExp however can match tokens longer than the 15 characters possible for identifier name, requiring yypushbacks.

Thus, is there a way to limit how many characters JFlex reads for a particular token?

Ambiguous Context-free grammar? / Shift/Reduce conflict in CUP

2021-05-03T23:59:02Z

I have the following context-free grammar for a simplified version of C++. When I run it with JFLEX and CUP I get a list of errors like that:

Warning : *** Reduce/Reduce conflict found in state #173
  between especificador ::= (*) 
  and     programa ::= (*) 
  under symbols: {VOID, CHAR, FLOAT, DOUBLE, SIGNED, UNSIGNED, INT, SHORT, LONG}
  Resolved in favor of the second production.

I think the problem is with instrucoesIf but I can't figure it out

start with programa;

programa ::= especificador tipo ID programa2 | DEFINE ID num CRLF programa | ; verificar depois o ERRO
especificador ::= AUTO | STATIC | EXTERN | CONST | ;
tipo ::= VOID | CHAR | FLOAT | DOUBLE | SIGNED inteiro | UNSIGNED inteiro | inteiro;

inteiro ::= SHORT | INT | LONG;
programa2 ::= SEMICOLON programa | LBRACK num RBRACK SEMICOLON programa | LPAREN listaParametros RPAREN bloco programa | COMMA listaID programa;

listaID ::= ID declaracaoParam2 listaIDTail;
listaIDTail ::= SEMICOLON | COMMA listaID;
listaParametros ::= listaParamRestante | ;
listaParamRestante ::= declaracaoParam declParamRestante;
declaracaoParam ::= tipo ID declaracaoParam2;
declaracaoParam2 ::= LBRACK num RBRACK | ;
declParamRestante ::= COMMA listaParamRestante | ;
bloco ::= LBRACE conjuntoInst RBRACE | SEMICOLON conjuntoInst ;
conjuntoInst ::= programa conjuntoInst | instrucoes conjuntoInst | ;
instrucoes ::= ID expressao SEMICOLON | RETURN expr SEMICOLON | PRINTF LPAREN expr RPAREN SEMICOLON | SCANF LPAREN ID RPAREN SEMICOLON | BREAK SEMICOLON | IF LPAREN expr RPAREN instrucoes instrucoesIf;

instrucoesIf ::= ELSE instrucoes | ;
expressao ::= atribuicao | LBRACK expr RBRACK atribuicao | LPAREN exprList RPAREN | ;
atribuicao ::= operadorAtrib expr;
operadorAtrib ::= EQ | MULTEQ | DIVEQ | MODEQ | PLUSEQ | MINUSEQ;
expr ::= exprAnd exprOr;
exprList ::= expr exprListTail | ;
exprListTail ::= COMMA exprList | ;
exprOr ::= OR exprAnd exprOr | ;
exprAnd ::= exprEqual exprAnd2;
exprAnd2 ::= AND exprEqual exprAnd2 | ;
exprEqual ::= exprRelational exprEqual2;
exprEqual2 ::= EQEQ exprRelational exprEqual2 | NOTEQ exprRelational exprEqual2 | ;
exprRelational ::= exprPlus exprRelational2;
exprRelational2 ::= LT exprPlus exprRelational2 | LTEQ exprPlus exprRelational2 | GT exprPlus exprRelational2 | GTEQ exprPlus exprRelational2 | ;

exprPlus ::= exprMult exprPlus2;
exprPlus2 ::= PLUS exprMult exprPlus2 | MINUS exprMult exprPlus2 | ;
exprMult ::= exprUnary exprMult2;
exprMult2 ::= MULT exprUnary exprMult2 | DIV exprUnary exprMult2 | ;
exprUnary ::= PLUS exprParenthesis | MINUS exprParenthesis | exprParenthesis;
exprParenthesis ::= LPAREN expr RPAREN | primary;
primary ::= ID primaryID | num | literal;
primaryID ::= LBRACK primary RBRACK | LPAREN exprList RPAREN | ;

literal ::= STRING | CHAR;
num ::= NUM_INT | NUM_FLOAT;

OCaml parser for ANTLR or JFlex

2021-04-14T15:25:23Z

I'm looking for grammar files of the OCaml language written in ANTLR4 or JFlex.

If they already exist somewhere, where I can find them? Else, how can I generate or write one of them by myself?

Is there an alternative to \b and/or negative lookahead for JFLEX?

2021-04-13T23:59:12Z

I am building a Scanner and can't seem to find a way to identify operators like "if" or "else" using JFlex & Regex. Since JFlex doesn't fully conform I can't use word-boundary or (?<=\s|^) + (?=\s|$) because neither ? or $ are allowed. The idea is to find the correctly written operators not ifo or elso. Thanks in advance.

Jflex get input filename

2021-04-03T17:34:02Z

In Jflex, how does one extract the input filename?

DisplayFilename.jflex:

%%

%class DisplayFilename

%eof{
    /* code to print the input filename goes here */
%eof}

%%

\n { /* do nothing */ }
. { /* do nothing */ }

Commands ran

jflex DisplayFilename
javac DisplayFilename.java
java DisplayFilename someinputfile.txt

desired output:

someinputfile.txt