Why I need to use NULL pointer when using strtok() in C?

Question

First my code:

#include <stdio.h>
#include <string.h>

int main()
{
    char str[100]="nahin bin kaysar";
    char *space;

    space = strtok(str," ");
    while(space != NULL) {
        space = strtok(NULL," ");
        puts(space);
    }
    return 0;
}

I know this program will split the string into three different parts. Before the loop I split the first word and assigned to the string space. But my question is why I had to useNULL pointer inside the loop as a parameter of the strtok() function. I have searched it for hours and no answer satisfied my needs.

Read the docs? (e.g. http://man7.org/linux/man-pages/man3/strtok.3.html) — Oliver Charlesworth, Apr 05 '17 at 17:31
This means continuing processing. Also `space = strtok(NULL," ");puts(space);` --> `puts(space);space = strtok(NULL," ");` — BLUEPIXY, Apr 05 '17 at 17:32
a) don't use `strtok()` in new code, there's a reason for [`strsep()`](http://man7.org/linux/man-pages/man3/strsep.3.html). b) What's unclear about `strtok()`? It internally stores tokenization context, so if you call it with `strtok(NULL, ...)`, it continues to operate on the most recent string. — dhke, Apr 05 '17 at 17:32
It is a confusing call because strtok() maintains internal state in statics - something that makes it difficult to understand how subsequent calls work at all, (unless in multithreaded code, in which case they don't:). — ThingyWotsit, Apr 05 '17 at 17:34
Your code is broken and it can crash, also, what part is unclear from [docs](https://www.tutorialspoint.com/c_standard_library/c_function_strtok.htm)? — BPL, Apr 05 '17 at 17:34
`NULL` informs `strtok()` to continue tokenizing from where it last stopped. A non-`NULL` value says to start anew. — chux - Reinstate Monica, Apr 05 '17 at 17:36
Beware confusion between "string", "array", and "pointer". `space` in your code is not a string: it's a pointer (pointing to (parts of) an array (each part being a string after strtok changed the array contents)). — pmg, Apr 05 '17 at 17:45

Sourav Ghosh · Answer 1 · 2017-04-05T17:57:51.683

Quoting C11, chapter §7.24.5.8, P4 (emphasis mine)

The strtok function then searches from there for a character that is contained in the current separator string. If no such character is found, the current token extends to the end of the string pointed to by s1, and subsequent searches for a token will return a null pointer. If such a character is found, it is overwritten by a null character, which terminates the current token. The strtok function saves a pointer to the following character, from which the next search for a token will start.

and P5,

Each subsequent call, with a null pointer as the value of the first argument, starts searching from the saved pointer [...]

So, this is a feature which enables strtok() to continue parsing the leftover of the initially supplied string.

To put it in other words, an input string, containing multiple instances of the delimiter can be parsed using a simple loop using this feature. For example, if a "sentence" is to be tokenized based on the "whitespace"s to separate out each "word" , then, after the first call, a simple loop, calling strtok() with NULL as first argument will make use of internal save pointer and require less explicit effort to manage the remainder of the actual sentence.

Without this feature, manual effort would have been required to keep a tab of the progress made in search of the delimiter.

score 0 · Answer 2 · edited May 23 '17 at 11:54

First of all, you should read again the docs and make sure you understand each parameter and the return.

Second, your code is broken, at certain point puts will use NULL as argument and it'll crash, you could fix it like this:

#include <string.h>
#include <stdio.h>

int main() {
    char str[100] = "nahin bin kaysar";
    const char s[2] = " ";
    char *token;

    token = strtok(str, s);
    printf("%s", token);
    while (token != NULL) {
        puts(token);
        token = strtok(NULL, s);
    }

    return 0;
}

In any case, if you have questions after reading carefully the documentation this other thread maybe can help you out.

Why I need to use NULL pointer when using strtok() in C?

2 Answers2