A better way to split a string into an array of strings in C/C++ using whitespace as a delimiter

Question

Sorry, my C/C++ is not that good, but the following existing code looks like garbage even to me. It also has a bug - fails when str = "07/02/2010" terminated by '\0' - . I think that instead of fixing a bug, it might as well be rewritten. In Python it is just 'kas\nhjkfh kjsdjkasf'.split(). I know this is C-ish code, but it cannot be that complicated to split a string! Sticking to the same signature, and without using extra libraries, how can I improve it - make it short and sweet? I can tell that this code smells, for instance because of the else clause all the way at the end.

LINE THAT FAILS:

_tcsncpy_s(
    s.GetBuffer((int) (nIndex-nLast)),
    nIndex-nLast,
    psz+nLast,
    (size_t) (nIndex-nLast)
);

With the string "07/02/2010" terminated by '\0' it will try to write 11 characters into a buffer that is only 10 characters long.

FULL FUNCTION:

#define 

// This will return the text string as a string array
// This function is called from SetControlText to parse the
// text string into an array of CStrings that the control
// Gadgets will attempt to interpret

BOOL CLVGridDateTimeCtrl::ParseTextWithCurrentFormat(const CString& str, const CGXStyle* pOldStyle, CStringArray& strArray )
{
    // Unused:
    pOldStyle;

    // we assume that the significant segments are seperated by space

    // Please change m_strDelim to add other delimiters

    CString s;

    LPCTSTR psz = (LPCTSTR) str;

    BOOL bLastCharSpace = FALSE;
    DWORD size = str.GetLength()+1;

    // (newline will start a new row, tab delimiter will
    // move to the next column).
    // parse buffer (DBCS aware)
    for (DWORD nIndex = 0, nLast = 0; nIndex < size; nIndex += _tclen(psz+nIndex))
    {
        // check for a delimiter
        if (psz[nIndex] == _T('\0') || _tcschr(_T("\r\n"), psz[nIndex]) || _tcschr(_T(" "), psz[nIndex])
            ||!_tcscspn(&psz[nIndex], (LPCTSTR)m_strDelim))
        {
            s.ReleaseBuffer();
            s.Empty();
            // abort parsing the string if next char
            // is an end-of-string
            if (psz[nIndex] == _T('\0'))
            {
                if (psz[nIndex] == _T('\r') && psz[nIndex+1] == _T('\n'))
                    nIndex++;

                _tcsncpy_s(s.GetBuffer((int) (nIndex-nLast)),
                    nIndex-nLast,
                            psz+nLast,
                            (size_t) (nIndex-nLast));
                CString temStr = s;
                strArray.Add(temStr);
                temStr.Empty();
                break;
            }

            else if (_tcscspn(&psz[nIndex], (LPCTSTR)m_strDelim) == 0 && !bLastCharSpace)
            {
                if (psz[nIndex] == _T('\r') && psz[nIndex+1] == _T('\n'))
                    nIndex++;

                _tcsncpy_s(s.GetBuffer((int) (nIndex-nLast)),
                    nIndex-nLast,
                            psz+nLast,
                            (size_t) (nIndex-nLast));
                CString temStr = s;
                strArray.Add(temStr);
                temStr.Empty();
                bLastCharSpace = TRUE;
                // abort parsing the string if next char
                // is an end-of-string
                if (psz[nIndex+1] == _T('\0'))
                    break;

            }
            // Now, that the value has been copied to the cell,
            // let's check if we should jump to a new row.
            else if (_tcschr(_T(" "), psz[nIndex]) && !bLastCharSpace)
            {
                if (psz[nIndex] == _T('\r') && psz[nIndex+1] == _T('\n'))
                    nIndex++;

                _tcsncpy_s(s.GetBuffer((int) (nIndex-nLast)),
                    nIndex-nLast,
                            psz+nLast,
                            (size_t) (nIndex-nLast));
                CString temStr = s;
                strArray.Add(temStr);
                temStr.Empty();
                bLastCharSpace = TRUE;
                // abort parsing the string if next char
                // is an end-of-string
                if (psz[nIndex+1] == _T('\0'))
                    break;
            }

            nLast = nIndex + _tclen(psz+nIndex);


        }
        else
        {   
            // nLast = nIndex + _tclen(psz+nIndex);
            bLastCharSpace = FALSE;
        }
    }
    if (strArray.GetSize())
        return TRUE;
    else
        return FALSE;
}

EDIT: m_strDelim = _T(","); and this member variable is used in this function only. I suppose I see the point of tokenization now - it tries to parse a date and time ... wait, there is more! Here is the code which calls this function below. Please help me improve this as well. Some of my co-workers claim that C# makes them no more productive than C++. I used to feel like an idiot for not being able to say the same about me.

// SetControlText will attempt to convert the text to a valid date first with
// the help of COleDateTime and then with the help of the Date control and the
// current format

BOOL CLVGridDateTimeCtrl::ConvertControlTextToValue(CString& str, ROWCOL nRow, ROWCOL nCol, const CGXStyle* pOldStyle)
{
    CGXStyle* pStyle = NULL;
    BOOL bSuccess = FALSE;

    if (pOldStyle == NULL)
    {
        pStyle = Grid()->CreateStyle();
        Grid()->ComposeStyleRowCol(nRow, nCol, pStyle);
        pOldStyle = pStyle;
    }

    // allow only valid input
    {
        // First do this
        CLVDateTime dt;

        if (str.IsEmpty())
        {
            ;
            // if (Grid()->IsCurrentCell(nRow, nCol))
            //  Reset();
            bSuccess = TRUE;
        }
        else if (dt.ParseDateTime(str,CLVGlobals::IsUSDateFormat()) && (DATE) dt != 0)
        {
            SetDateTime(dt);
            if (m_bDateValueAsNumber)
                str.Format(_T("%g"), (DATE) dt);
            else
                str = dt.Format();
            bSuccess = TRUE;
        }
        else
        {
            // parse the string using the current format
            CStringArray strArray;
            if (!ParseTextWithCurrentFormat(str, pOldStyle, strArray))
                return FALSE;

            UpdateNullStatus(m_TextCtrlWnd);

            SetFormat(m_TextCtrlWnd, *pOldStyle);

            int nArrIndex = 0;
            for(int i=0; i<m_TextCtrlWnd.m_gadgets.GetSize(); i++)
            {
                int val = m_TextCtrlWnd.m_gadgets[i]->GetValue();   
                // s.Empty();
                if(m_TextCtrlWnd.m_gadgets[i]->IsKindOf(RUNTIME_CLASS(SECDTNumericGadget)))
                {
                    // TRACE(_T("The value %s\n"), strArray[nArrIndex]);
                    ((CLVDTNumericGadget*)m_TextCtrlWnd.m_gadgets[i])->m_nNewValue = _ttoi(strArray[nArrIndex]);    
                    nArrIndex++;
                    if (nArrIndex>strArray.GetUpperBound())
                            break;
                }
                else if(m_TextCtrlWnd.m_gadgets[i]->IsKindOf(RUNTIME_CLASS(SECDTListGadget)) && val!=-1)
                {
                    int nIndex = ((CLVDTListGadget*)m_TextCtrlWnd.m_gadgets[i])->FindMatch(strArray[nArrIndex], ((CLVDTListGadget*)m_TextCtrlWnd.m_gadgets[i])->GetValue()+1);
                    if (nIndex!=-1)
                    {
                        // TRACE(_T("The value %s\n"), strArray[nArrIndex]);
                        ((CLVDTListGadget*)m_TextCtrlWnd.m_gadgets[i])->SetValue(nIndex);
                        nArrIndex++;
                        if (nArrIndex>strArray.GetUpperBound())
                            break;
                    }

                }

                CLVDBValue dbDate = m_TextCtrlWnd.GetDateTime();
                if (dbDate.IsNull())
                    str = _T("");
                else
                {
                    CLVDateTime dt = (CLVDateTime)dbDate;
                    if (m_bDateValueAsNumber)
                        str.Format(_T("%g"), (DATE) dt);
                    else
                        str = dt.Format();
                }
            }
            bSuccess = TRUE;
        }
    }

    if (pStyle)
        Grid()->RecycleStyle(pStyle);

    return bSuccess;
}

Answer 1

The String Toolkit Library (Strtk) has the following solution to your problem:

#include <string>
#include <deque>
#include "strtk.hpp"
int main()
{ 
   std::string data("kas\nhjkfh kjsdjkasf");
   std::deque<std::string> str_list;
   strtk::parse(data, ", \r\n", str_list);
   return 0;
}

More examples can be found Here

Answer 2

In C++, it's probably easiest to use a stsringstream:

std::istringstream buffer("kas\nhjkfh kjsdjkasf");

std::vector<std::string> strings;

std::copy(std::istream_iterator<std::string>(buffer),
          std::istream_iterator<std::string>(),
          std::back_inserter(strings));

I haven't tried to stick to exactly the same signature, mostly because most of it is non-standard, so it doesn't apply to C++ in general.

Another possibility would be to use Boost::tokenizer, though obviously that does involve another library, so I won't try to cover it in more detail.

I'm not sure if that qualifies as "bizarro syntax" or not. I may have to work a bit on that part...

Edit: I've got it -- initialize the vector instead:

std::istringstream buffer("kas\nhjkfh kjsdjkasf");

std::vector<std::string> strings(
    (std::istream_iterator<std::string>(buffer)),
    std::istream_iterator<std::string>());

The "bizarro" part is that without the extra parentheses around the first argument, this would invoke the "most vexing parse", so it would declare a function instead of defining a vector. :-)

Edit2: As far as the edit to the question goes, it seems nearly impossible to answer directly -- it depends on too many types (e.g., CGXStyle, CLVDateTime) that are neither standard nor explained. I, for one, can't follow it in any detail at all. Offhand, this looks like a fairly poor design, letting the user enter things that are more or less ambiguous, and then trying to sort out the mess. Better to use a control that only allows unambiguous input to start with, and you can just read some fields that contain a date and time directly.

Edit3: code to do the splitting that also treats commas as separators could be done like this:

#include <iostream>
#include <locale>
#include <algorithm>
#include <vector>
#include <sstream>

class my_ctype : public std::ctype<char> {
public:
    mask const *get_table() { 
        // this copies the "classic" table used by <ctype.h>:
        static std::vector<std::ctype<char>::mask> 
            table(classic_table(), classic_table()+table_size);

        // Anything we want to separate tokens, we mark its spot in the table as 'space'.
        table[','] = (mask)space;

        // and return a pointer to the table:
        return &table[0];
    }
    my_ctype(size_t refs=0) : std::ctype<char>(get_table(), false, refs) { }
};

int main() { 
    // put our data in a strea:
    std::istringstream buffer("first kas\nhjkfh kjsdjk,asf\tlast");

    // Create a ctype object and tell the stream to use it for parsing tokens:
    my_ctype parser;
    buffer.imbue(std::locale(std::locale(), &parser));

    // separate the stream into tokens:
    std::vector<std::string> strings(
        (std::istream_iterator<std::string>(buffer)),
        std::istream_iterator<std::string>());

    // copy the tokes to cout so we can see what we got:
    std::copy(strings.begin(), strings.end(), 
        std::ostream_iterator<std::string>(std::cout, "\n"));
    return 0;
}

Answer 3

The best way to do it would be using strtok. That link should be self explanatory on how to use it, and you can use multiple delimiters as well. Very handy C function.

Answer 4

Quite an over the top way of sorting this problem is to use the Qt libraries. If you're using KDE then they're installed already. The QString class has a member function split which works like the python version. For example

QString("This is a string").split(" ", QString::SkipEmptyParts)

returns a QStringList of QStrings:

["This", "is", "a", "string"]

(in pythonic syntax). Note the second argument is required or else should the words be split by multiple spaces, each individual one would be returned.

In general I find with the help of the Qt libraries, most of the simplicity of python, eg. simple string parsing and list iteration, can be handled with ease and with the power of C++.

Answer 5

Parsing strings in C/C++ rarely turns out to be a simple matter. The method you posted looks like it has quite a bit of "history" involved in it. For example, you state that you want to split the string on white space. But the method itself appears to be using a member variable m_strDelim as part of the splitting decision. Simply replacing the method could lead to other unexpected issues.

Using an existing tokenizing class such as this Boost library could simplify things quite a bit.

Answer 6

You can use boost::algorithm::split. I.e.:

std::string myString;
std::vector<std::string> splitStrings;
boost::algorithm::split(splitStrings, myString, boost::is_any_of(" \r\n"));

Answer 7

A better method than my other answer: TR1's regex feature. Here's a small tutorial to get you started. This answer is C++, uses regular expressions (which is perhaps the best / easiest way to split a string), and I used it myself recently, so I know it's a nice tool.