How to store temp value for strtok without changing the initial token

Question

Here is part of my code:

int main ()
{
char *sentence;
char *token;

int counter = 1;
sentence = (char*)malloc(255*sizeof(char));

scanf("%[^\n]s", sentence);
token = strtok(sentence, " ");

char *temp;

while(token != NULL)
{
    printf("Token %d: %s\n", counter, token);
    token = strtok(NULL, " ");
    //temp = token;
    //temp = strtok(NULL, " ");

    counter++;
}
return 0;
}

If I run this when i type: "why herrow there" it gives me:

Token 1: why

Token 2: herrow

Token 3: there

If I uncomment the temp then it only gives me:

Token1: why

Token 2: herrow

It seems even tho I think I didn't hinder my token value with temp, it still affects my token value. I don't want temp to have any affect on my original token. How do I do that?

side note, son't use `scanf("%[^\n]s", sentence);` read [Reading a line using `scanf()` not good?](http://stackoverflow.com/questions/17294809/reading-a-line-using-scanf-not-good/17294869#17294869) — Grijesh Chauhan, Sep 29 '13 at 04:51
if i uncomment the temp = token; temp = strtok(NULL, " "); it still gives the same output as written as above... (awk, he deleted the comment) — LarsChung, Sep 29 '13 at 04:57
I think I misunderstood your question so I deleted comment,, let me check again. wait.. — Grijesh Chauhan, Sep 29 '13 at 04:59
And what do you expect?temp is a pointer to char, token is a pointer to char. You set them to point to the same place in memory and expect that they will point to different place in memory.That's all. — zubergu, Sep 29 '13 at 08:02

Grijesh Chauhan · Accepted Answer · 2013-09-29T06:38:06.703

1

You string has three words "why herrow there" The case when you add temp statements:

Step first:

token = strtok(sentence, " ");   <-- sentence: `"why\0herrow there"` 
                                 // token = sentence
char *temp;

first iteration:

while(token != NULL)  // token is not null <-------------------------------+
{                                                                          | 
    printf("Token %d: %s\n", counter, token); // first time print why      |
                                                                           |
    token = strtok(NULL, " ");  <-- sentence: `"why\0herrow\0there"`       |//step-2
                                <-- token points to "herrow" substring  (*)|
    temp = token;               <---temp = token                           |
    temp = strtok(NULL, " ");   <---sentence: `"why\0herrow\0there"`       |//step-3
                                <-- temp = "there" sub string              |//Last token
    counter++;                             |-------------------------------+
}

second iteration of while loop:

while(token != NULL) // token is not null, it is pointing to sustring "herrow"
{
    printf("Token %d: %s\n", counter, token); printing "herrow"
    token = strtok(NULL, " "); <-- no next token, token becomes NULL //step-4
    temp = token;    
    temp = strtok(NULL, " ");  <-- no next token, so temp becomes NULL //step-5

    counter++;
}

Third iteration token is NULL

While loop breaks!

So does it only prints:

Token1: why

Token 2: herrow

Based on comment!

token = strtok(sentence, " "); // first token
next_token = token;
while(next_token != NULL){
    printf("Token %d: %s\n", counter, token);
    if(next_token = strtok(NULL, " "))
           token = next_token;   //Last token in string
    // here you have last token that is not NULL 
    counter++;
}
// next_token is NULL, but token is not NULL it is equals to last token in string
counter--;
printf("Token %d: %s\n", counter, token);

Code working.

edited Sep 29 '13 at 06:38

answered Sep 29 '13 at 05:10

Grijesh Chauhan

57,103
20
141
208

I set token as a different string, temp, but it still affects my token value. Why does it do that isn't temp and and token supposed to be different string? Basically, how do I make them independent of each other – LarsChung Sep 29 '13 at 05:13
@LarsChung Didn't get you, `token` and `temp` becomes NULL at different function call of `strtok()` in second iteration. Both are independent. Note `strtok()` returns NULL when it doesn't find new token. Read each comment and steps – Grijesh Chauhan Sep 29 '13 at 05:31
Let me rephrase: I strtok token (not temp). Meaning now it is at herrow. I set temp = token. Meaning temp is now the same string as token. I strtok temp (not token), but in the second iteration, token is also NULL, when it should be at there. i.e. why is token at NULL and not there at second iteration when I only strtok token once in the first iteration? – LarsChung Sep 29 '13 at 05:37
So what I basically want to know is how to fix this so that temp and token are independent and doesn't get affected by each other by using strtok... – LarsChung Sep 29 '13 at 05:40
@LarsChung See first Notice output you got two printf output so you loop executes two times. Correct? Now before while loop you call `strtok()` first time and in while loop body you call `strtok()` two times so in two iteration you call strtok `1 + (2 + 2)` = **`5`** times. -- read answer precisely I commented where token and temp becomes NULL in last calls of `strtok()`. Let me know your expected output. – Grijesh Chauhan Sep 29 '13 at 05:42
I want the output to be the Token 1: why, Token 2: herrow,Token 3: there, even with the temp. I just want to use temp as a checking string if next token is NULL or not. If it isn't, I want to go back to the token which hasn't been affected by temp using strtok. – LarsChung Sep 29 '13 at 05:48
To make it more clear, I'm trying to make a fsm for a cmd line interpreter, and I want temp to be a prechecker if I should to another state or end the state there. If I see if it has to go through another function, I need my original token before it is hindered by strtok on temp. – LarsChung Sep 29 '13 at 05:51
@LarsChung wait let me give few minutes to think over it. – Grijesh Chauhan Sep 29 '13 at 06:05
If it's just a NFA, it's no problem, but I have some states that will go to another state through empty string, which makes me figure out how to check the next token before I do the function. – LarsChung Sep 29 '13 at 06:09
Maybe this is more simpler question. Can you code so that I can check if next token is NULL and if it is, undo ur action? Basically if you strtok to check, can u go back to the previous state? (it could also be another variable that holds the previous state). – LarsChung Sep 29 '13 at 06:30
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/38258/discussion-between-larschung-and-grijesh-chauhan) – LarsChung Sep 29 '13 at 06:34

alk · Answer 2 · 2013-09-29T10:16:58.680

You can achieve what you want by using the reentrant version of strtok() that is strtok_r():

#define _POSIX_SOURCE

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

int main(void)
{ 
  char * sentence = malloc(255 * sizeof(*sentence));

  scanf("%[^\n]s", sentence);

  {
    int counter = 0;
    char * sp1 = NULL;
    char * token = strtok_r(sentence, " ", &sp1);

    while (NULL != token)
    {
      counter++;

      printf("Token %d: %s\n", counter, token);
      token = strtok_r(NULL, " ", &sp1);

      {
        char * sp2 = sp1;
        char * temp = strtok_r(NULL, " ", &sp2);
        if (NULL != temp)
        {
          printf("Temp %d: %s\n", counter, temp);
          temp[strlen(temp)] = ' ';
        }
      }
    }
  }

  return 0;
}

Note: This only work if the set of delimiters passed to strtok() only contains one characters (' ' in this example).

How to store temp value for strtok without changing the initial token

2 Answers2