atom feed13 messages in [JavaCC] Parser doesn't recognize...
FromSent OnAttachments
SatyamApr 18, 2005 6:13 pm 
Jerry SchwarzApr 18, 2005 6:38 pm 
SatyamApr 19, 2005 8:46 am 
SatyamApr 19, 2005 9:26 am 
Sreenivas ViswanadhaApr 19, 2005 9:38 am 
SatyamApr 21, 2005 10:08 am 
Sreenivas ViswanadhaApr 21, 2005 10:13 am 
Sreenivas ViswanadhaApr 21, 2005 10:21 am 
Nathan D. RyanApr 21, 2005 10:30 am 
sre...@viswanadha.netApr 21, 2005 11:46 am 
SatyamApr 21, 2005 2:35 pm 
sre...@viswanadha.netApr 21, 2005 4:28 pm 
SatyamApr 22, 2005 1:10 am 
Subject:Re: [JavaCC] Parser doesn't recognize string literals in BNF when in a lexical state other than DEFAULT
From:Satyam (
Date:Apr 19, 2005 8:46:27 am

Unfortunately no, 40 is the left parenthesis, space is 32 or 0x20. Anyway, I do have a:

<PHP> SKIP : { " " | "\t" | "\n" | "\r"

skiping also all three types of comments available in PHP. (I admit, though, that you made me look just in case)

Moreover, the spaces didn't change when I replaced the "(" with <LPAREN> and it did work doing that. (failing later on when it got to the ";" at the end of the statement, and it did work when I defined it as <SEMICOLON>)

My guess is that literal strings taken from the parser are included as if defined under the <DEFAULT> lexical state and thus unavailable in other lexical states.

When I defined <LPAREN> as a <PHP> TOKEN:, then it was explicitly declared under the PHP lexical state, but when defined within the parser, it seems it doesn't go there. I don't really know, it is just my guess. If it is so then it is a bug, as literal strings used in the parser should be available either in the current lexical state, which might be hard to do, or in all lexical states.

In the meantime, is there any way to start the grammar in a lexical state other than <DEFAULT>? Because if it is so, what I could do is go to a named lexical state at the start, when I am expecting plain HTML, which I simply let pass through, and go back to <DEFAULT> for the PHP part, which is the one I need to analyze. Can I use SwitchTo in, say TOKEN_MGR_DECLS or should I extend the standard token manager class and add that in the constructor?


----- Original Message ----- From: "Jerry Schwarz" <> To: <> Sent: Tuesday, April 19, 2005 3:38 AM Subject: Re: [JavaCC] Parser doesn't recognize string literals in BNF when in a lexical state other than DEFAULT

I think 40 is a space and I think what you're seeing is a problem with skipping white space. Presumably you have a <PHP> SKIP: .... somewhere. I'm guessing that because your grammar has the "(" here javacc isn't doing any more token scanning after it sees the <INCLUDE> and the parse is failing. Whether this is a bug or not isn't clear to me.

Satyam wrote:

I have the following sections in my .jj file:

<PHP> TOKEN : { <REQUIRE_ONCE: "require_once"> | <REQUIRE: "require"> | <INCLUDE_ONCE: "include_once"> | <INCLUDE: "include"> | .... several others ... }


void IncludeStatement() : {} { ( <INCLUDE> | <INCLUDE_ONCE> | <REQUIRE> | <REQUIRE_ONCE>) "(" String() ")" ";" }

The debug trace says:

Call: IncludeStatement Consumed token: <"include"> <PHP>Current character : ( (40) No string literal matches possible. Starting NFA to match one of : { ... several .... } <PHP>Current character : ( (40) Return: IncludeStatement

I returns in error, having the "(" there, clearly specified in the grammar.

Nevertheless, if I add:

<PHP> TOKEN: { <LPAREN: "("> | <RPAREN: ")"> }

And change the "(" and ")" to LPAREN and RPAREN in the definition for IncludeStatement it reaches the ";" without error, but now complains about the ";"

What is going on? Literals in the parser are supposed to work. Does it have anything to do with being within a state other than DEFAULT? Unfortunately a PHP program starts with HTML grammar until reaching the first <?php, and that is my DEFAULT lexical state and when finding the <?php I switch to the PHP lexical state.