C++: Extract Tokens from String using only STL

Question

I want to extract tokens from a given fraction expression, which is passed as a string,

Fraction s("-5 * 7/27");

Where the extracted tokens hold either a single operator or a sequence of digits as an operand, using the stl container only.

May someone guide me to do so? I want to know how to extract the tokens and differentiate the operand from operator, thank you.

Possible duplicate: [How do I tokenize a string in C++?](http://stackoverflow.com/questions/53849/how-do-i-tokenize-a-string-in-c). — Jesse Good, Jun 04 '12 at 03:43
I checked, they use delimiter as a space, but in my case wont help. You think is it good to used string::operator[] to get each char and evaluate it? — newbieLinuxCpp, Jun 04 '12 at 03:52
You can also try: [A string tokenizer in C++ that allows multiple separators](http://stackoverflow.com/questions/2654114/a-string-tokenizer-in-c-that-allows-multiple-separators). — jxh, Jun 04 '12 at 03:57
It depends. Can it be assumed that there will be spaces between the tokens? — Daniel Hanrahan, Jun 04 '12 at 04:02
Yes there are spaces, I need to extract the tokens and push it into a queue — newbieLinuxCpp, Jun 04 '12 at 04:05
Just use `boost::tokenizer`, it will make this a lot easier. — Jesse Good, Jun 04 '12 at 04:34

Daniel Hanrahan · Accepted Answer · 2012-06-04T04:56:38.360

Assuming there are always spaces between the tokens, here's how put them all in a queue:

#include <string>
#include <queue>
#include <sstream>
#include <iostream>

using namespace std;

int main()
{
    stringstream formula("-5 * 7/27");
    string a_token;
    queue<string> tokens; 

    // Use getline to extract the tokens, and then push them onto the queue.
    while (getline(formula, a_token, ' ')) {
         tokens.push( a_token );
    }

    // Print and pop each token.
    while (tokens.empty() == false) {
        cout << tokens.front() << endl;
        tokens.pop();
    }
}

Running the program will output:

-5
*
7/27

Now to determine which are operators, numbers, or fractions, you can do something like this in the loop:

    if (a_token == "+" || a_token == "-" || a_token == "*" || a_token == "/")
    {
        // It's an operator
        cout << "Operator: " << a_token << endl;
    }
    else
    {
        // Else it's a number or a fraction.

        // Now try find a slash '/'.
        size_t slash_pos = a_token.find('/');

        // If we found one, it's a fraction.
        if (slash_pos != string::npos) {
            // So break it into A / B parts.

            // From the start to before the slash.
            string A = a_token.substr(0, slash_pos);

            // From after the slash to the end.
            string B = a_token.substr(slash_pos + 1);

            cout << "Fraction: " << A << " over " << B << endl;
        }
        else
        {
            // Else it's just a number, not a fraction.
            cout << "Number: " << a_token << endl;
        }
    }

This website: http://www.cplusplus.com/reference/string/string/ will give you information on the string functions.

After modifying the code and running it again, you'll get output like this:

Number: -5
Operator: *
Fraction: 7 over 27

Yes sir, I did this way, but it is not useful for my case because I need to separate operands from operators, like this: - 5 * 7 / 27 — newbieLinuxCpp, Jun 04 '12 at 04:23
Oh I believe I follow now. You want to be able to split the fraction parts too. I'll update the post in a minute to show you how. — Daniel Hanrahan, Jun 04 '12 at 04:26
No problem. You'll probably want to convert the strings into numbers later if you're using them in calculations. You can do it like this: atoi(A.c_str()). (Don't forget to upvote the answer!) — Daniel Hanrahan, Jun 04 '12 at 05:03
I am sorry am bothering. I am curious, what if I have no spaces in my expression, in this case separating by space is not useful. what can I do instead? say ("1-10*1/(5+2)+9") — newbieLinuxCpp, Jun 04 '12 at 14:50
That's a lot more difficult. You'd need to build a lexical analyzer / parser. There's lots of info online for these. This might help you also: http://www.ibm.com/developerworks/library/j-w3eval/index.html — Daniel Hanrahan, Jun 04 '12 at 15:14

C++: Extract Tokens from String using only STL

1 Answers1