What is the best way to tokenize a string in C++?

Question

I was trying using strtok(), but it is giving segmentation faults. Could anyone please tell me where the problem is in the code and is there any better way to tokenize a string other than strtok() ?

void tokenize(char *tagToFind, char *record, char *delim)
{
    char *token;
    char *itr;
    char *tag;
    char *tag5;
    int toBreak=0;
    token = strtok(record,delim);
    while (token != NULL)
    {
            itr = token;
            while (*itr != '{')
            {
                    tag = itr;
                    itr++;
                    tag++;
            }
            tag = '\0';
            if ((strcmp(tag, tagToFind) == 0))
                    break;
            else
                    token = strtok(NULL,delim);
    }

    if(strcmp(tag5, "tag5") == 0)
    {
            cout<<"\n\n\n\n\t\ttag5 is present.";
    }
}

int main()
{
    char *tag = "tag5";
    char *record = "tag1{0}|tag2{0}|tag3{0}|tag4{0}|tag5{tag51{0};tag52{0};tag53{0};tag54{0};tag55{tag551{0}:tag552{0}:tag553{0}:tag554{0}:tag555{0}}}";
    char *delim = "|";
    tokenize(tag, record, delim);
    return 0;
}

http://stackoverflow.com/questions/236129/how-to-split-a-string-in-c — sujin, Feb 05 '14 at 13:10
Is this C or C++? If it's the latter, then you should check out std::string — olevegard, Feb 05 '14 at 13:11
Segmentation fault is because you are modifying string literal pointed by `record`. `strtok()` modifies first argument read [strtok causing segfault but not when step through code](http://stackoverflow.com/questions/17551665/strtok-causing-segfault-but-not-when-step-through-code/17551779#17551779) — Grijesh Chauhan, Feb 05 '14 at 13:16
@crashmstr It is more C code indeed. edit: opps title says c++ — Grijesh Chauhan, Feb 05 '14 at 13:19

Simple · Accepted Answer · 2014-02-05T13:18:59.457

3

char const* const tag = "tag5";
char const* const record = "tag1{0}|tag2{0}|tag3{0}|tag4{0}|tag5{tag51{0};tag52{0};tag53{0};tag54{0};tag55{tag551{0}:tag552{0}:tag553{0}:tag554{0}:tag555{0}}}";
char const delim = '|';

std::stringstream ss(record);
for (std::string token; std::getline(ss, token, delim); ) {
    // Handle token here.
}

Example here.

edited Feb 05 '14 at 13:18

answered Feb 05 '14 at 13:13

Simple

13,992
2
47
47

@Mariners should click on right if accepts this answer, and *don't* forgot reading @ [JohnBode's answer](http://stackoverflow.com/a/21578521/1673391). – Grijesh Chauhan Feb 05 '14 at 13:29

score 1 · Answer 2 · answered Feb 05 '14 at 13:20

You're getting a segfault because you're using strtok on a string literal. Remember that strtok modifies the input string (it replaces all instances of the delimiters with 0), and modifying a string literal results in undefined behavior; on some platforms (such as yours, apparently), string literals are stored in a read-only memory segment, hence the error.

Your code shouid work with the following change:

char record[] = "tag1{0}|tag2{0}|tag3{0}|tag4{0}|tag5{tag51{0};tag52{0};tag53{0};tag54{0};tag55{tag551{0}:tag552{0}:tag553{0}:tag554{0}:tag555{0}}}";

Instead of record being a pointer to a string literal, it's now an array of char that can be modified by your code.

Having said all that, if you're using C++, Simple's solution is probably a better way to go.

What is the best way to tokenize a string in C++?

2 Answers2