strlen not counting newlines?

Question

I embedded lua into my project and came across a strange (for me) behavior of strlen and lua interpreting. I was trying to load a string, containing lua code, with luaL_loadbuffer and it consistently threw error of "unexpected symbol" on whatever was the last line of the lua code, except if the whole chunk was written in one line. so for example:

  function start()
      print("start")
  end

would always results error: unexpected symbol on 3rd line, but

function start() print("start") end

loads successfully.

I figured out that loading the same chunk with luaL_loadstring, gives no errors, and saw that it uses strlen to determine the length of the specified string (I used std::string::size) and also that using strlen to provide the length of the string to the luaL_loadbuffer also results in successful loading.

Now the question was: what may be the difference between strlen and std::string::size, and at my most surprise the answer is that strlen is not counting new lines ('\n'). That is:

 const char* str  = "this is a string\nthis is a newline";
  std::string str2(str);
  str2.size(); // gives 34
  strlen(str); // gives 33

The difference between the size, and the value returned by strlen was always the number of new line characters.

My questions are:

Does strlen really not counting newlines or I am missing something?
How do newlines affect the interpretation of the lua code internally?

I am using vs 2015 and lua 5.3.0

EDIT:

My first example was not exact, and did not produce the detailed effect for me neither, but I was able to recreate the problem from the original code:

std::fstream _Stream("test.lua", std::ios::ate | std::ios::in);
    std::string _Source;
    if(_Stream.is_open()) {
        _Source.resize(_Stream.tellg());
        _Stream.seekg(0, std::ios::beg);
        _Stream.read(&_Source[0], _Source.size());
        _Stream.close();
    }

    std::cout << "std::string::size() = " << _Source.size() << std::endl;
    std::cout << "strlen() = " << strlen(_Source.c_str()) << std::endl;

The content of test.lua is "function start()\n\tprint("start")\nend\n\nstart()"

The difference is the number of newlines:

https://i.stack.imgur.com/y0QOW.png

i believe that strlen dont give you the final character (the hidden \0), and the size function is called after the string constructor is called (i.e the final character is included in str2, but not in the lenght gave by strlen function) — Jordy Baylac, Aug 30 '15 at 23:19
so what?? so what about str2.size(); // gives 34 strlen(str); // gives 33 — Jordy Baylac, Aug 30 '15 at 23:22
why you say that cout << strlen(str) << endl; gives 34 in a place and in another place you say 33? — Jordy Baylac, Aug 30 '15 at 23:25
You are mistaken - both expressions [evaluate to 34](http://rextester.com/JMOJB66242) — Igor Tandetnik, Aug 31 '15 at 00:12
@IgorTandetnik my example was not exact, please see edited version — zaw, Aug 31 '15 at 07:44

ryanpattison · Accepted Answer · 2015-08-31T10:07:44.877

1

Window's line endings (CR+LF) are two characters making the file size larger than the number of characters in the string so the resize operation uses the file size and not the length of the null-terminated string. strlen reports the length of the null-terminated string and counts \n as a single character. You can make the size match the length of the C string by resizing the string to match afterwards:

_Source.resize(strlen(_Source.c_str()) + 1);

edited Aug 31 '15 at 10:07

answered Aug 31 '15 at 10:02

ryanpattison

6,151
1
21
28

related: http://stackoverflow.com/questions/2641639/fstreams-tellg-seekg-returning-higher-value-than-expected – ryanpattison Aug 31 '15 at 10:17
Thanks, saving the file with unix line-endings shows the difference. – zaw Aug 31 '15 at 10:25

strlen not counting newlines?

1 Answers1