How can I read the MNIST dataset with C++?

Question

I was reading the following question:

and there was some C++ code for reading the MNIST database. Upon trying it I found out that it worked fine until the place where it started reading the data.

which is the following code:

 for(int i=0;i<number_of_images;++i)
      {
        for(int r=0;r<n_rows;++r)
        {
            for(int c=0;c<n_cols;++c)
            {
                unsigned char temp=0;
                file.read((char*)&temp,sizeof(temp));
                //cout<<(int)temp<<" "; //printing the pixel in integer format

            }
        }
    }

I tried printing out the integer value of the variable "temp" however I didn't get the correct number for the pixels(all of them were zero). I'm not sure what's wrong there, each pixel takes one bytes space and then I convert it to an int and it doesn't work. Why does this happen? thank you in advance

just an update, the code is perfectly fine. I'm not sure why it didn't work last night while I was checking it, however today that I woke up I tried running it again and the outputs were correct. Maybe there was some kind of conflict yesterday on my computer and a simple reboot solved the problem? I'm not sure, however after studying the format of these files I couldn't simply understand why this code wouldn't work, it didn't make sense to me which is exactly the reason why I rerun it. Thankfully everything is OK right now! — ksm001, Oct 21 '12 at 08:32

score 5 · Answer 1 · edited Aug 16 '17 at 14:17

5

I had the same problem, the first few bytes of data being fine and then the rest was filled with 0s. It turns out that the file stream was failing when it first encountered byte 0x01A. The fix for this was changing:

ifstream file ("data");

to

ifstream file ("data", std::ios::binary);

edited Aug 16 '17 at 14:17

SurvivalMachine

7,946
15
57
87

answered May 23 '13 at 02:19

Nyph

51
1
2

David Campbell · Accepted Answer · 2012-11-29T01:55:31.713

When working with the MNIST data set, I had the same problem that you had. I could read the labels, but the training and test set images were mostly bogus; the training set was filled almost entirely with 175, and the testing set was filled almost entirely with 0s (except for the first 6 images). Rebooting did not fix the problem and I was unable to determine why the file reading was not working correctly.

For anyone with this same problem, I would suggest instead using the data files located at http://cis.jhu.edu/~sachin/digit/digit.html. The data is already organized by number (no label/image association required), and the arrays of pixel values are simply encoded one after the other. Knowing that each array is 28x28 and that there are 1000 images for each number, you can easily write code to input the individual image arrays of pixel values.

If anyone is looking for simple .csv files, I posted the train and test set on my website, you can get them [here](http://pjreddie.com/projects/mnist-in-csv/). Hope they are useful! — pjreddie, Dec 04 '13 at 05:06

Jayhello · Answer 3 · 2018-09-19T13:12:55.630

Below is complete code to read mnist and convert it to cv Mat:

uint32_t swap_endian(uint32_t val) {
    val = ((val << 8) & 0xFF00FF00) | ((val >> 8) & 0xFF00FF);
    return (val << 16) | (val >> 16);
}

void read_mnist_cv(const char* image_filename, const char* label_filename){
    // Open files
    std::ifstream image_file(image_filename, std::ios::in | std::ios::binary);
    std::ifstream label_file(label_filename, std::ios::in | std::ios::binary);

    // Read the magic and the meta data
    uint32_t magic;
    uint32_t num_items;
    uint32_t num_labels;
    uint32_t rows;
    uint32_t cols;

    image_file.read(reinterpret_cast<char*>(&magic), 4);
    magic = swap_endian(magic);
    if(magic != 2051){
        cout<<"Incorrect image file magic: "<<magic<<endl;
        return;
    }

    label_file.read(reinterpret_cast<char*>(&magic), 4);
    magic = swap_endian(magic);
    if(magic != 2049){
        cout<<"Incorrect image file magic: "<<magic<<endl;
        return;
    }

    image_file.read(reinterpret_cast<char*>(&num_items), 4);
    num_items = swap_endian(num_items);
    label_file.read(reinterpret_cast<char*>(&num_labels), 4);
    num_labels = swap_endian(num_labels);
    if(num_items != num_labels){
        cout<<"image file nums should equal to label num"<<endl;
        return;
    }

    image_file.read(reinterpret_cast<char*>(&rows), 4);
    rows = swap_endian(rows);
    image_file.read(reinterpret_cast<char*>(&cols), 4);
    cols = swap_endian(cols);

    cout<<"image and label num is: "<<num_items<<endl;
    cout<<"image rows: "<<rows<<", cols: "<<cols<<endl;

    char label;
    char* pixels = new char[rows * cols];

    for (int item_id = 0; item_id < num_items; ++item_id) {
        // read image pixel
        image_file.read(pixels, rows * cols);
        // read label
        label_file.read(&label, 1);

        string sLabel = std::to_string(int(label));
        cout<<"lable is: "<<sLabel<<endl;
        // convert it to cv Mat, and show it
        cv::Mat image_tmp(rows,cols,CV_8UC1,pixels);
        // resize bigger for showing
        cv::resize(image_tmp, image_tmp, cv::Size(100, 100));
        cv::imshow(sLabel, image_tmp);
        cv::waitKey(0);
    }

    delete[] pixels;
}

Usage:

string base_dir = "/home/xy/caffe-master/data/mnist/";
string img_path = base_dir + "train-images-idx3-ubyte";
string label_path = base_dir + "train-labels-idx1-ubyte";

read_mnist_cv(img_path.c_str(), label_path.c_str());

Output as below:

How can I read the MNIST dataset with C++?

3 Answers3

Linked