Tokenize a String in C++

Question

I have a string currentLine="12 23 45"

I need to extract 12, 23, 45 from this string without using Boost libraries. Since i am using string, strtok fails for me. I have tried a number of things still no success.

Here is my last attempt

while(!inputFile.eof())
    while(getline(inputFile,currentLine))
        {
            int countVar=0;
            int inputArray[10];
            char* tokStr;
            tokStr=(char*)strtok(currentLine.c_str()," ");

            while(tokstr!=NULL)
            {
            inputArray[countVar]=(int)tokstr;
            countVar++;
            tokstr=strtok(NULL," ");
            }
        }
}

the one without strtok

string currentLine;
while(!inputFile.eof())
    while(getline(inputFile,currentLine))
        {
            cout<<atoi(currentLine.c_str())<<" "<<endl;
            int b=0,c=0;
            for(int i=1;i<currentLine.length();i++)
                {
                    bool lockOpen=false;
                    if((currentLine[i]==' ') && (lockOpen==false))
                        {
                        b=i;
                        lockOpen=true;
                        continue;
                        }
                    if((currentLine[i]==' ') && (lockOpen==true))
                        {
                        c=i;
                        break;
                        }
                }
            cout<<b<<"b is"<<" "<<c;    
        }

Is that your real code, with the case-mismatched `tokstr` and `tokStr`? Also, http://stackoverflow.com/questions/236129/how-to-split-a-string-in-c — Mr Lister, Apr 17 '12 at 10:28
You don't report how they fail. Do they compile? Do they crash? Do they give incorrect results? — David Schwartz, Apr 17 '12 at 10:30
yeah tokstr must be copied while i was undoing things... sorry for typo — CodeMonkey, Apr 17 '12 at 10:36

k06a · Accepted Answer · 2012-04-17T10:41:57.010

9

Try this:

#include <sstream>

std::string str = "12 34 56";
int a,b,c;

std::istringstream stream(str);
stream >> a >> b >> c;

Read a lot about c++ streams here: http://www.cplusplus.com/reference/iostream/

edited Apr 17 '12 at 10:41

answered Apr 17 '12 at 10:29

k06a

17,755
10
70
110

error: variable 'std::istringstream stream' has initializer but incomplete type – CodeMonkey Apr 17 '12 at 10:38
@user1290495, `#include ` – hmjd Apr 17 '12 at 10:41

score 5 · Answer 2 · answered Apr 17 '12 at 10:28

std::istringstream istr(your_string);

std::vector<int> numbers;
int number;
while (istr >> number)
    numbers.push_back(number);

Or, simpler (though not really shorter):

std::vector<int> numbers;
std::copy(
    std::istream_iterator<int>(istr),
    std::istream_iterator<int>(),
    std::back_inserter(numbers));

(Requires the standard headers <sstream>, <algorithm> and <iterator>.)

score 0 · Answer 3 · answered Apr 17 '12 at 10:35

You can also opt for Boost tokenizer ......

#include <iostream>
#include <string>
#include <boost/foreach.hpp>
#include <boost/tokenizer.hpp>
using namespace std;
using namespace boost;

int main(int argc, char** argv)
{
    string str= "India, gold   was dear";
    char_separator<char> sep(", ");
    tokenizer< char_separator<char> > tokens(str, sep);
    BOOST_FOREACH(string t, tokens)
    {
        cout << t << "." << endl;
    }
}

"without using Boost libraries" – k06a Apr 17 '12 at 10:44 — k06a, Apr 17 '12 at 10:44

score 0 · Answer 4 · answered Apr 17 '12 at 10:47

stringstream and boost::tokenizer are two possibilities. Here is a more explicit solution using string::find and string::substr.

std::list<std::string>
tokenize(
  std::string const& str,
  char const token[])
{
  std::list<std::string> results;
  std::string::size_type j = 0;
  while (j < str.length())
  {
    std::string::size_type k = str.find(token, j);
    if (k == std::string::npos)
      k = str.length();

    results.push_back(str.substr(j, k-j));
    j = k + 1;
  }
  return results;
}

Hope this helps. You can easily turn this into an algorithm that writes the tokens to arbitrary containers or takes a function handle that processes the tokens.

Tokenize a String in C++

4 Answers4