Unable to print all the tokenized string

Question

I'm trying to write a code which tokenize the string 2 times. In 1st pass it tokenize based on "&" and on second pass it should tokenize based on "|". But after 1st pass, it does not continue. Below is sample code.

#include <iostream>
#include <string.h>
            
using namespace std;
            
int main () {
             
    char value[] = "Hello | my | Name & is | Pratik & lets see";
    char *token = strtok (value, "&");
 
    while (token != NULL) {
        cout << "\n token: " << token;

        char *token_1 = strtok (token, "|");
        while (token_1 != NULL) {
                  
            cout << "\n token_1: " << token_1;
            token_1 = strtok (NULL, "|");
        }
        token = strtok (NULL, "&");
    }
    return 0;
}

I'm getting out put token: Hello | my | Name token_1: Hello token_1: my token_1: Name

I'm I missing anything?

First thing to do: [edit] and format and indent your code properly. — Jabberwocky, Oct 13 '21 at 12:32
Also tell us what output you expect and what output you get. Don't answer with a comment but [edit] the question. — Jabberwocky, Oct 13 '21 at 12:34
`strtok` cannot be used in this fashion. It uses a static char buffer internally. Thus you cannot jump back and forth using `strtok` on the same string using a different delimiter. You're using C++ -- there is no need to use `strtok` to accomplish your goal. There are much, much better ways in C++ to do this task. — PaulMcKenzie, Oct 13 '21 at 12:35
Related: [do it in C++ not in C](https://stackoverflow.com/a/55680/1387438). — Marek R, Oct 13 '21 at 13:19

score 0 · Answer 1 · answered Oct 13 '21 at 12:38

strtok has only enough brainpower to keep track of just one string being tokenized.

The shown code uses two nested strtok loops, with what's effectively two strings, and expects each loop's strtok to be independent of each other, and handle only its own string. Unfortunately strtok does not work this way. Calling strtok with a NULL string value results in the most recent strtok-ed string continuing to get tokenized.

However, the solution to your conundrum is very simple: replace strtok with strtok_s, and provide the appropriate state pointer to each instance.

Unable to print all the tokenized string

1 Answers1