Tokenize a string to separate values in C++

Question

I have a string of the form

ADD,R1,#5

I want to convert it to three separate variables

string s1 = "ADD";

int register = 1; [after removing the 'R' and converting to int]

int literal = 5; [after removing the # and converting to int]

I tried using strok() function by converting it into char* but the output doesn't seem tight

Use: `std::string`, `std::string::find()` and `std::string::substr`. I also recommend `std::istringstream` for converting text number to internal representation. — Thomas Matthews, Oct 11 '17 at 22:58
You could use `std::getline(data_file, text_string, ',')` to read all the text up to a comma into a `text_string` std::string variable. — Thomas Matthews, Oct 11 '17 at 23:00
"int literal = 1; [after removing the 5 and converting to int]" Seems a bit odd that you are discarding the last `5`. Isn't this a split by `,` ? — Isac, Oct 11 '17 at 23:05

score 1 · Accepted Answer · answered Oct 11 '17 at 23:43

You want to write a parser. To keep my answer really simple and short, I wrote a very very simple parser for you. Here is it:

// Written by 'imerso' for a StackOverflow answer

#include <string.h>
#include <stdio.h>
#include <stdlib.h>


// Extract next substring from current pointer
// to next separator or end of string
char* NextToken(char** cmd, const char* sep)
{
    char* pStart = *cmd;
    char* pPos = strstr(*cmd, sep);

    if (!pPos)
    {
        // no more separators, return the entire string
        return *cmd;
    }

    // return from the start of the string up to the separator
    *pPos = 0;
    *cmd = pPos+1;
    return pStart;
}


// Warning: this is a very simple and minimal parser, meant only as an example,
// so be careful. It only parses the simple commands without any spaces etc.
// I suggest you to try with ADD,R1,#5 similar commands only.
// A full parser *can* be created from here, but I wanted to keep it really simple
// for the answer to be short.
int main(int argc, char* argv[])
{
    // requires the command as a parameter
    if (argc != 2)
    {
        printf("Syntax: %s ADD,R1,#5\n", argv[0]);
        return -1;
    }

    // the command will be split into these
    char token[32];
    int reg = 0;
    int lit = 0;

    // create a modificable copy of the command-line
    char cmd[64];
    strncpy(cmd, argv[1], 64);
    printf("Command-Line: %s\n", cmd);

    // scan the three command-parameters
    char* pPos = cmd;
    strncpy(token, NextToken(&pPos, ","), 32);
    reg = atoi(NextToken(&pPos, ",")+1);
    lit = atoi(NextToken(&pPos, ",")+1);

    // show
    printf("Command: %s, register: %u, literal: %u\n", token, reg, lit);

    // done.
    return 0;
}

Tokenize a string to separate values in C++

1 Answers1