Preprocess a custom text file to remove comments using boost spirit

Question

I've text which contains "equations" like :

-- This is comment
ABC:= 121-XY1/7 > 45 OR SS >= 3
    ZY2 AND -- This is another comment
    (JKL * PQR) < 75;

JKL:= PP1 OR 
      PP2/2 XOR PP3;

ZZ_1:=A-B > 0 XOR (B2 % GH == 6 AND 
   SP == 4
-- Again a comment
    NOT AX > GF < 2 OR C*AS2 >= 5);

I decided to use boost spirit to parse this text , as of now I just need to know my operands and operators.

I've referred this nice answer (thanks to sehe :) ) to write my expression grammar (relational operators not written yet)

However I cannot strip my comments with :-

     qi::phrase_parse(input.begin()
     ,input.end()
     ,p >> ';' // parser object
     ,qi::space | "--" >> *(qi::char_ - qi::eol) >> qi::eol
     ,result //expression object,  (boost::variant with boost::recursive_wrapper)
     );

because it gives several errors, some of the post says to tweak a boost header file.

So I'm using another grammar to strip off comment first using :

     qi::phrase_parse(input.begin()
     ,input.end()
     ,qi::char_ >> *qi::char_ 
     , qi::space | "--" >> *(qi::char_ - qi::eol) >> qi::eol
     ,stripped // std::string 
     );

But this gives me a text with all space and comment removed :

ABC:=121-XY1/7>45ORSS>=3ZY2AND(JKL*PQR)<75;JKL:=PP1ORPP2/2XORPP3;ZZ_1:=A-B>0XOR(B2%GH==6ANDSP==4NOTAX>GF<2ORC*AS2>=5);

So, the question is how can I strip just the comments, preserving space and newlines ?

^{Using : Boost Version 1.55.0}

I am processing a text file that I also need to strip out comments (C# not C++) and I simply do it with a Regex to zap everything after and including the comment marker. That way the structure of the file remains. I do this step before I parse my file. Whereas something like spirit is meant to carve the file up into tokens isn't it? — Peter M, Feb 01 '14 at 18:03
@PeterM yeah, with python/perl regex it will be a matter of few lines, but I kinda interested in _"compiler-like"_ functionality. Like correct evaluation of expressions, pin-point the errors, etc — P0W, Feb 01 '14 at 18:07
I'm thinking you need to think like a C/C++ preprocessor rather than parser/compiler at this step. — Peter M, Feb 01 '14 at 18:09
Is [this](http://coliru.stacked-crooked.com/a/d4ca41240892c52e) what you want? — llonesmiz, Feb 01 '14 at 19:10
About your initial situation... How are you declaring your skipper type in your parser object? If the code you pasted is what you are using, you would need to use `decltype` or `BOOST_TYPEOF`. If that were your problem (and I'm taking a shot in the dark) I believe the usual solution is creating a skipper grammar (something like [this](http://stackoverflow.com/questions/8521314/custom-skip-parser-with-boostspirit)). — llonesmiz, Feb 01 '14 at 19:18
@cv_and_he Thanks ! And I don't know why I was not able to see output with `phrase_parse(f,l,qi::char_ >> *qi::char_ ,"--" >> *(qi::char_ - qi::eol) >> qi::eol,strip);`, (until I redirected the output to a text file), which is basically the same which you told, if I'm correct ?. Please make that an answer, I'll accept. Also regarding the initial solution, could you please tell how to define grammar for relation operator, similar to [this](http://stackoverflow.com/a/8468822/1870232) ? and also incorporate the skipper to skip the comment. I tried the above post but it didn't work — P0W, Feb 01 '14 at 19:26

score 2 · Accepted Answer · edited May 23 '17 at 11:49

2

Your exact code sample is missing. Allow me to add a sample skipper to that "boolean expression grammar" you linked to:

std::string const input = 
        "a and\n"
        "-- abacadabra\n"
        "b;";

typedef std::string::const_iterator It;

// ADDED: allow comments
qi::rule<It> skip_ws_and_comments 
    = qi::space 
    | "--" >> *(qi::char_-qi::eol) >> qi::eol
    ;

parser<It, qi::rule<It> > p;

That's all the changes required. Output:

result: (a & b)

See it Live On Coliru

edited May 23 '17 at 11:49

Community

1
1

answered Feb 01 '14 at 19:44

sehe

374,641
47
450
633

Perhaps a c++03 version might come in handy: http://coliru.stacked-crooked.com/a/aaaa0971c0b3fed2 – sehe Feb 01 '14 at 20:47
Thanks ! I was waiting for you :D My [code](http://coliru.stacked-crooked.com/a/8866e92369b728bb) , from all your answers :) Any suggestions/doc/answers for grammar for relational operators similar to airthmetic operators ? – P0W Feb 01 '14 at 20:47
@P0W Just generalize 'make_binop' to recognize more than single char tokens: http://coliru.stacked-crooked.com/a/37faa9423bb47bbc (oh and some cleanup. Rule #1: clear code makes a clear mind) – sehe Feb 01 '14 at 21:27
Thanks a ton ! I've just started learning boost spirit, wanted to have it in one of my project and your awesome answers are very helpful. – P0W Feb 01 '14 at 21:40
1

Thanks! Keep in mind I conveniently ignored operator precedence for now. See e.g. [this answer](http://stackoverflow.com/questions/20387627//20387628#20387628), this [one that links to more resources](http://stackoverflow.com/a/12622808/85371), and [this one that actually refactors one of the samples that also does relational operators with precedence](http://stackoverflow.com/a/19715064/85371) – sehe Feb 01 '14 at 21:57

Preprocess a custom text file to remove comments using boost spirit

1 Answers1