How to separate strings based on tokens

Question

#include <iostream>
using namespace std; 

void
main()
{

    string target_str =  "1.2.3.4:3333 servertype=simics,arch=x86"; 
    string host_str; 
    string port_str; 
    string type_str; 
    string arch_str; 

    host_str = target_str.substr(0, target_str.find_first_of(':'));
    port_str = target_str.substr(target_str.find_first_of(':')+1);
    type_str  = target_str.substr(target_str.find_first_of(':'));
    arch_str = target_str.substr(target_str.find_first_of(':'));
}

On completion I want the following values:

host_str = 1.2.3.4, 
port_str = 3333, 
type_str = simics 
arch_str = x86.

Yes regex works:

std::string target_str =  "1.2.3.4:3333 servertype=simics,arch=x86"; 
string host_str; 
string port_str; 
string type_str; 
string arch_str; 

regex expr("([\\w.]+):(\\d+)\\s+servertype\\s*=\\s*(simics|openocd)(?:\\s*,\\s*| )arch=(x86|x64)"); 
smatch match; 
if (regex_search(target_str, match, expr)) 
{ 
    cout << "host: " << match[1].str() << endl; 
    cout << "port: " << match[2].str() << endl; 
    cout << "type: " << match[3].str() << endl; 
    cout << "arch: " << match[4].str() << endl; 
}

But unfortunately this program has to compile on Windows and Lunix so hence I have to use std strings only

http://stackoverflow.com/questions/5804453/c-regular-expressions-with-boost-regex — Ian Medeiros, Jun 02 '14 at 18:15
research the usage of `regexp` - your problem is a strait forward case for `regexp`. — NirMH, Jun 02 '14 at 18:17
Instead of describing, you should edit your post to show exactly what a sample input string will consist of. So far, I don't see a clear example that shows the IP along with the other data in a single string. — PaulMcKenzie, Jun 02 '14 at 18:19
@ChrisTarazi Don't forget to actually answer the question, comments are not answers. — Owl_Prophet, Jun 02 '14 at 18:26
"But unfortunately this program has to compile on Windows and Linux so hence I have to use std strings only" - I don't any reason why regex would not compile on any of those two platforms. — Csq, Jun 02 '14 at 19:30
If you can use c++11, just employ the new regex functionality. — rwols, Jun 02 '14 at 21:04

score 0 · Answer 1 · answered Jun 02 '14 at 18:22

Here is a code that tokenizes an input given by a std::string and outputs the tokens in a std::vector<string>. You can specify multiple delimiters (in a std::string).

#include <iostream>
#include <vector>
#include <string>
#include <stdexcept> 

std::vector<std::string> 
split(const std::string& str, const std::string& delim){
    std::vector<std::string> result;
    if (str.empty())
        throw std::runtime_error("Can not tokenize an empty string!");
    std::string::const_iterator begin, str_it;
    begin = str_it = str.begin(); 
    do {
        while (delim.find(*str_it) == std::string::npos && str_it != str.end())
            str_it++; // find the position of the first delimiter in str
        std::string token = std::string(begin, str_it); // grab the token
        if (!token.empty()) // empty token only when str starts with a delimiter
            result.push_back(token); // push the token into a vector<string>
        while (delim.find(*str_it) != std::string::npos && str_it != str.end())
            str_it++; // ignore the additional consecutive delimiters
        begin = str_it; // process the remaining tokens
        } while (str_it != str.end());
    return result;
}

int main() {
    std::string test_string = ".this is.a.../.simple;;test;;;END";
    std::string delim = "; ./"; // string containing the delimiters
    std::vector<std::string> tokens = split(test_string, delim);           
    for (std::vector<std::string>::const_iterator it = tokens.begin(); 
        it != tokens.end(); it++)
            std::cout << *it << std::endl;
}

Thanks it looks very nice but when I run it, it throws up an exception. Debug Assertion failed. Expression String iterator not derefrencable. C:\Windows....\include\xstring line 79 — Santhosh Kumar, Jun 02 '14 at 18:54
Have no idea why you're getting this. I can compile and run it without any problem on OS X (using g++). Are you using the code above identically? Try to replace `const_iterator` with `iterator` — vsoftco, Jun 02 '14 at 18:56
VSoftCo thanks for your code but since I have found a easier way out I shall not attempt your method for now. i shall use strtok. Thanks again — Santhosh Kumar, Jun 02 '14 at 19:17

score 0 · Answer 2 · answered Jun 02 '14 at 18:25

0

I believe what you want is strtok(), as said in the comments below. Be sure to do your research, as it can be a squirrely function, and if you don't know what you're doing, you can break things. It will do exactly what you want. It can take a list of tokens to break up a string, and then give them to you one at a time.

answered Jun 02 '14 at 18:25

Owl_Prophet

362
5
15

I am from the C world and would have gladly used strtok() but that Std Strings do not support it right? I don't have the freedom to change the definition of the 5 strings. – Santhosh Kumar Jun 02 '14 at 18:51
You can use the .c_str() method of the Std string class. Does that work? I do not know the extent of your situation. – Owl_Prophet Jun 02 '14 at 18:57
You can convert to a `char*` by following [this](http://stackoverflow.com/a/16502000/2193236) instead of dealing with `const char*` and the `.c_str()` function. – ctzdev Jun 02 '14 at 18:59
@ChrisTarazi I guess that might work, but there is a built in method for it, as I posted above. – Owl_Prophet Jun 02 '14 at 19:00
Fantastic this works!! Thanks a lot all. I will post the full function in a while bit, – Santhosh Kumar Jun 02 '14 at 19:17

Santhosh Kumar · Answer 3 · 2014-06-02T21:00:34.033

#include <iostream>
#include <string>
using namespace std; 

void

main()

{

        std::string target_str =  "1.2.3.4:3333 servertype=simics, arch=x86"; 
        string host_str; 
        string port_str; 
        string type_str; 
        string arch_str; 
        char *token;
        char *myString = new char [target_str.length()+1];
        strcpy (myString, target_str.c_str());
        token = strtok(myString, ":= ");
        host_str.assign(token, strlen(token));
        token = strtok(NULL, ":= ");
        port_str.assign(token, strlen(token));
        token = strtok(NULL, ":= ");
        token = strtok(NULL, ":= ");
        type_str.assign(token, strlen(token));
        token = strtok(NULL, ":= ");
        token = strtok(NULL, ":= ");
        arch_str.assign(token, strlen(token));
        cout << "host: " << host_str;
        cout << " port: " << port_str;
        cout << " type: " << type_str;
        cout << " arch: " << arch_str;
}

host: 1.2.3.4 port: 3333 type: simics, arch: x86

Thanks to all this works!!

This adds the comma to the servertype, you might want to add a `,` in there — Mooing Duck, Jun 02 '14 at 22:04

How to separate strings based on tokens

3 Answers3