Why this yacc+lex basic parser does not handle CONTROL+D / EOF?

Question

I have a yacc/lex program to handle this kind of lines (in this example it just handles one format, but the idea is that it will obviously handle more formats):

% cat test.csv 
20191201 170003296,1.102290,1.102470,0
20191201 170004413,1.102320,1.102470,0
20191201 170005270,1.102290,1.102470,0
20191201 170006063,1.102280,1.102460,0
20191201 170006629,1.102260,1.102440,0
20191201 170007523,1.102410,1.102470,0
20191201 170007573,1.102410,1.102530,0
20191201 170035268,1.102490,1.102530,0
20191201 170036505,1.102490,1.102540,0
20191201 170043219,1.102490,1.102530,0

The lex lexical analyzer (lexer.l):

%{

#include <time.h>
#include "grammar.h"

void read_float_number(void);
void read_integer_number(void);
void read_date_YYYYMMDD_HHMMSSmmm(void);
void yyerror(const char* msg);

%}

%%    
                                                                                                /* YYYYMMDD HHMMSSmmm DATE */
[12][09][0-9][0-9][0-1][0-9][0-3][0-9][ ][0-2][0-9][0-5][0-9][0-5][0-9][0-9][0-9][0-9]          { read_date_YYYYMMDD_HHMMSSmmm(); return DATETIME; }

                                                                                                /* FLOAT NUMBER */
[0-9]+\.[0-9]+                                                                                  { read_float_number(); return FLOAT_NUMBER; }

                                                                                                /* INTEGER NUMBER */
[0-9]+                                                                                          { read_integer_number(); return INTEGER_NUMBER; }

                                                                                                /* PASS ',' CHARACTER */
,                                                                                               { return ','; } 

                                                                                                /* PASS '\n' CHARACTER */
\n                                                                                              { return '\n'; } 

                                                                                                /* PASS UNEXPECTED CHARACTER */
.                                                                                               { return yytext[0]; }


%%

/* READ FLOAT NUMBER */
void read_float_number(void) {
        printf("void read_float_number(void)\n");
        printf("#%s#\n", yytext);
        sscanf(yytext, "%lf", &yylval.float_number);
        printf("%lf\n", yylval.float_number);
}

/* READ INTEGER NUMBER */
void read_integer_number(void) {
        printf("void read_integer_number(void)\n");
        printf("#%s#\n", yytext);
        sscanf(yytext, "%ld", &yylval.integer_number);
        printf("%ld\n", yylval.integer_number);
}

/* READ YYYYMMDD HHMMSSmmm DATE */
void read_date_YYYYMMDD_HHMMSSmmm(void) {

        printf("void read_date_YYYYMMDD_HHMMSSmmm(void)\n");
        printf("#%s#\n", yytext);

        /*  DATETIME STRUCT TM */
        struct tm dt;

        /* READ VALUES */
        sscanf(yytext, "%4d%2d%2d %2d%2d%2d", &dt.tm_year, &dt.tm_mon, &dt.tm_mday, &dt.tm_hour, &dt.tm_min, &dt.tm_sec);

        /* NORMALIZE VALUES */
        dt.tm_year = dt.tm_year - 1900;         /* NORMALIZE YEAR */
        dt.tm_mon = dt.tm_mon - 1;              /* NORMALIZE MONTH */
        dt.tm_isdst = -1;                       /* NO INFORMATION ABOUT DST */
        mktime(&dt);                            /* NORMALIZE STRUCT TM */

        /* PRINT DATE TIME */
        char buffer[80];
        strftime(buffer, 80, "%c %Z", &dt);
        printf("%s\n", buffer);

        /* COPY STRUCT TM TO YACC RETURN VALUE */
        memcpy(&yylval.datetime, &dt, sizeof(dt));


}

The yacc grammar (grammar.y):

%{

#include <time.h>
#include <stdio.h>

%}

%union {

        struct tm       datetime;               /* DATE TIME VALUES */
        double          float_number;           /* 8 BYTES DOUBLE VALUE */
        long            integer_number;         /* 8 BYTES INTEGER VALUE */

}

%token  <datetime>              DATETIME
%token  <float_number>          FLOAT_NUMBER
%token  <integer_number>        INTEGER_NUMBER

%%

input:                          /* empty */
                        | input lastbid_lastask

lastbid_lastask:        DATETIME ',' FLOAT_NUMBER ',' FLOAT_NUMBER ',' INTEGER_NUMBER '\n'      { printf("MATCH %lf %lf %ld\n", $3, $5, $7); }
                        ;

%%

extern FILE *yyin;

int main(int argc, char *argv[]) {

        while(!feof(yyin)) {
                yyparse();
        }
        return 0;

}

The makefile:

% cat makefile 
CCFLAGS = -std=c89 -c
YFLAGS = -d     # Forces generation of y.tab.h
OBJS = lexer.o grammar.o
TARGET = readfile

readfile:               $(OBJS)
                        cc $(OBJS) -std=c89 -ll -o $(TARGET)

grammar.h grammar.o:    grammar.y
                        yacc $(YFLAGS) -ogrammar.c grammar.y
                        cc $(CCFLAGS) grammar.c

lexer.o:                lexer.l grammar.h
                        lex -olexer.c lexer.l
                        cc $(CCFLAGS) lexer.c

clean:
                        rm -f $(OBJS) grammar.[ch] lexer.c

Now I compile the program and there are no errors, but when I try to execute it I get this:

% cat test.csv | ./readfile
Segmentation fault (core dumped)

Now if I replace:

while(!feof(yyin)) 

with:

while(1) 

Then I get this:

% cat test.csv | ./readfile
void read_date_YYYYMMDD_HHMMSSmmm(void)
#20191201 170003296#
Sun Dec  1 17:00:03 2019 CET
void read_float_number(void)
#1.102290#
1.102290
void read_float_number(void)
#1.102470#
1.102470
void read_integer_number(void)
#0#
0
MATCH 1.102290 1.102470 0
void read_date_YYYYMMDD_HHMMSSmmm(void)
#20191201 170004413#
Sun Dec  1 17:00:04 2019 CET
void read_float_number(void)
#1.102320#
1.102320
void read_float_number(void)
#1.102470#
1.102470
void read_integer_number(void)
#0#
0
...

So it works, but the program does not end with the EOF. While I know a core dump can mean many things, what could I do to further locate the issue and get a normal behaviour?

Answer 1

Don't call yyparse() in a loop. It will parse the entire input and return; when it returns you know the entire input has been parsed (or a syntax error was encountered). There should be no need for any EOF testing.

(There are isolated cases in which you need to break this rule, most of which have to do with either the scanner returning end of input indicators other than at the end of the input, or the parser using YYACCEPT/YYABORT in orderly to prematurely terminate the parse. In other words, if you have a case where you need to break this rule, you already knew you would have to do that.)

while (!feof(file)) {…} has a whole FAQ entry explaining why it's almost always a bug. (Summary: the EOF flag is set after a read detects EOF, so the fact that EOF is not set before you do the read proves nothing. The while(!feof(file)) idiom pretty well guarantees that at the end of the file you'll get an unexpected EOF -- unexpected in the sense of "But I just checked for EOF...".)

I don't think the FAQ covers this particular issue, though, which is specific to programs using (f)lex. When a (f)lex scanner hits the end of file, it sets yyin to NULL. Then, if yywrap tells it that there is no more input, yylex returns 0, which tells its caller (yyparse) that the end of file was reached. Then yyparse finishes the parse and returns. If you then loop, yyin is NULL, and feof(NULL) is Undefined Behaviour. That's why your program segfaulted.

When you remove the feof test (but still loop), you reenter yyparse, but this time with yyin set to NULL. The flex scanner takes that to mean "use the default input", i.e. stdin. If yyin was previously some input file, that means that the new invocation of yyparse will try to get its input from the terminal, which is probably not what you expected. On the other hand, if it was stdin which reached EOF, then you'll just be in a hard loop, continuously receiving new EOF signals from stdin.