use caffe to train Lenet with CSV data

Question

Excuse me, I have a question on using caffe for hd data? I try to run an example on the Kaggle mnist csv data with the following steps

1. use h5py to convert it to h5 data. (I use the caffe-example.py to convert)

2. Then modify the lenet_train_test_prototxt and train it. I am quite at a loss of this step.

The only change I made here is

layer {
  name: "mnist"
  type: "HDF5Data"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  transform_param {
    scale: 0.00390625
  }
  data_param {
    source: "data/mnist_train_h5.txt"
    batch_size: 64
  }
}

how to change the lenet_train_test_prototxt to suit the data? Or also there are some other files I need to change? The error log is

enF0724 18:21:11.052737 79373 hdf5_data_layer.cpp:76] Check failed: !this->layer_param_.has_transform_param() HDF5Data does not transform data.

> *** Check failure stack trace: ***
>     @     0x7fe8188bbdaa  (unknown)
>     @     0x7fe8188bbce4  (unknown)
>     @     0x7fe8188bb6e6  (unknown)
>     @     0x7fe8188be687  (unknown)
>     @     0x7fe818caec10  caffe::HDF5DataLayer<>::LayerSetUp()
>     @     0x7fe818c520a3  caffe::Net<>::Init()
>     @     0x7fe818c53e12  caffe::Net<>::Net()
>     @     0x7fe818c0ba20  caffe::Solver<>::InitTrainNet()
>     @     0x7fe818c0c9c3  caffe::Solver<>::Init()
>     @     0x7fe818c0cb96  caffe::Solver<>::Solver()
>     @           0x40c8f0  caffe::GetSolver<>()
>     @           0x406541  train()
>     @           0x404a81  main
>     @     0x7fe817dcdec5  (unknown)
>     @           0x40502d  (unknown)
>     @              (nil)  (unknown) Aborted (core dumped)ter code here

Answer 1

I assume you have one hdf5 data file 'data/mnist_train_h5.hd5'.

1. As you can see from the error message you got, "HDF5Data" layer does not support data transformation. Specifically, you cannot scale the data by the layer.
   Thus, any transformations you wish to have, you must apply them yourself during the creation of 'data/mnist_train_h5.hd5'.

2. "HDF5Data" layer does not accept data_param, but rather hdf5_data_param with a parameter source specifying a list of hd5 binary files. In your case you should prepare an extra text file 'data/mnist_train_h5.txt' with a single line:

data/mnist_train_h5.hd5

This text file will tell caffe to read 'data/mnist_train_h5.hd5'.

The resulting layer should look like:

layer {
  name: "mnist"
  type: "HDF5Data"
  top: "data"
  top: "label"
  hdf5_data_param {
    source: "data/mnist_train_h5.txt"
    batch_size: 64
  }
  include {
    phase: TRAIN
  }
}