1

I am writing grammar for multi line strings (Text Block) in java. The delimiter for the start and end of a text block is triple quotes. I can successfully parse and build the AST for the text blocks and its content, except for one issue: the TEXT_BLOCK_START token is being returned after the tokens from the second lexer. I am using this as a guide: flow diagram. According to the ANTLR2 documentation, the way that I have implemented this should produce the desired token stream:

TEXT_BLOCK_START -> content from second lexer, etc... -> TEXT_BLOCK_END

I have tried changing the order of the action and the delimiter, the order of the rules, and using select() instead of selector.push().

Here are the important parts of the main class:

final Lexer lexer = new Lexer(reader);
lexer.setCommentListener(contents);

final Lexer secondLexer =
                new Lexer(lexer.getInputState());

lexer.setTokenObjectClass("antlr.CommonHiddenStreamToken");
secondLexer.setTokenObjectClass("antlr.CommonHiddenStreamToken");

final TokenStreamHiddenTokenFilter filter = new 
    TokenStreamHiddenTokenFilter(lexer);

final TokenStreamSelector selector = new TokenStreamSelector();
lexer.selector = selector;
secondLexer.selector = selector;
selector.addInputStream(filter, "filter");
selector.addInputStream(secondLexer, "secondLexer");
selector.select(filter);

The lexer (main lexer) rule:

TEXT_BLOCK_START
    :   "\"\"\"" {selector.push("secondLexer");}
    ;

The secondary lexer rule:

TEXT_BLOCK_END
   : "\"\"\"" {selector.pop();}
   ;

As stated above, everything parses as expected, except that the token stream looks like this:

content from second lexer, etc... -> TEXT_BLOCK_END -> TEXT_BLOCK_START

What am I missing here?

Nick Mancuso
  • 171
  • 6
  • 1
    I don’t think there are many people here able to help you with such an old version. If possible, use ANTLR 4 instead. – Bart Kiers Jul 03 '20 at 05:26
  • @BartKiers thanks for the reply. Unfortunately, at this time, ANTLR2 is a requirement. Is there a better place to ask for help? I've also posted this question to the ANTLR mailing list. – Nick Mancuso Jul 03 '20 at 06:02
  • 1
    I've not been a frequent visitor on the mailinglist, but I think you can expect an answer there sooner than here. Many of the (little bit) more complicated ANTLR 2 questions here go unanswered, is my experience. Anyway, best of luck! – Bart Kiers Jul 03 '20 at 06:47

0 Answers0