35

I am relatively new to machine learning/python/ubuntu.

I have a set of images in .jpg format where half contain a feature I want caffe to learn and half don't. I'm having trouble in finding a way to convert them to the required lmdb format.

I have the necessary text input files.

My question is can anyone provide a step by step guide on how to use convert_imageset.cpp in the ubuntu terminal?

Thanks

Shai
  • 111,146
  • 38
  • 238
  • 371
pwhc
  • 516
  • 1
  • 6
  • 16

1 Answers1

64

A quick guide to Caffe's convert_imageset

Build

First thing you must do is build caffe and caffe's tools (convert_imageset is one of these tools).
After installing caffe and makeing it make sure you ran make tools as well.
Verify that a binary file convert_imageset is created in $CAFFE_ROOT/build/tools.

Prepare your data

Images: put all images in a folder (I'll call it here /path/to/jpegs/).
Labels: create a text file (e.g., /path/to/labels/train.txt) with a line per input image . For example:

img_0000.jpeg 1
img_0001.jpeg 0
img_0002.jpeg 0

In this example the first image is labeled 1 while the other two are labeled 0.

Convert the dataset

Run the binary in shell

~$ GLOG_logtostderr=1 $CAFFE_ROOT/build/tools/convert_imageset \
    --resize_height=200 --resize_width=200 --shuffle  \
    /path/to/jpegs/ \
    /path/to/labels/train.txt \
    /path/to/lmdb/train_lmdb

Command line explained:

  • GLOG_logtostderr flag is set to 1 before calling convert_imageset indicates the logging mechanism to redirect log messages to stderr.
  • --resize_height and --resize_width resize all input images to same size 200x200.
  • --shuffle randomly change the order of images and does not preserve the order in the /path/to/labels/train.txt file.
  • Following are the path to the images folder, the labels text file and the output name. Note that the output name should not exist prior to calling convert_imageset otherwise you'll get a scary error message.

Other flags that might be useful:

  • --backend - allows you to choose between an lmdb dataset or levelDB.
  • --gray - convert all images to gray scale.
  • --encoded and --encoded_type - keep image data in encoded (jpg/png) compressed form in the database.
  • --help - shows some help, see all relevant flags under Flags from tools/convert_imageset.cpp

You can check out $CAFFE_ROOT/examples/imagenet/convert_imagenet.sh for an example how to use convert_imageset.

Graham
  • 7,431
  • 18
  • 59
  • 84
Shai
  • 111,146
  • 38
  • 238
  • 371
  • Hi, thanks for the great guide. I took a slightly different route and edited the _create_imagenet_ script. However upon running it I got the follow error; i.e. **" a total of 0 images"** Creating train lmdb... I0715 16:54:06.121748 4120 convert_imageset.cpp:79] Shuffling data I0715 16:54:06.122463 4120 convert_imageset.cpp:82] A total of 0 images. I0715 16:54:06.123065 4120 db.cpp:34] Opened lmdb /home/pwhc/caffe/GPRLearn/lmdb/GPR_train_lmdb Any thoughts? – pwhc Jul 15 '15 at 16:23
  • @pwhc it seems like no image files were found. check the path to images and the image names in the labels gile – Shai Jul 15 '15 at 16:29
  • when reference in the train and test text files i included the file trype; as in **$DATA/train.txt** when it should have simply been **$DATA/train**. Appreciate the guidance – pwhc Jul 15 '15 at 17:19
  • how can you have all your images in a single folder and yet still have a train/test text file? where does the train/test split take place then? – pwhc Sep 23 '15 at 14:34
  • 1
    @pwhc you need to construct **different** lmdb/leveldb for train/test. Therefore, you need **two** different files `/path/to/labels/train.txt` and `/path/to/labels/test.txt` the image names in those files should be different, but they can point to images in the same or in different folders - it's up t o you to organize them. – Shai Sep 24 '15 at 05:23
  • so the lmdb train/test databases are defined by the text files? – pwhc Sep 24 '15 at 14:50
  • 1
    @Shai best Tutorial for caffe! Thanks – black Mar 13 '16 at 09:31
  • Can you please explain why first image is 1 and the other two are 0? you mean there is only 1 positive image and the other two are negative images? what if I have positives in one folder and negatives in another folder? – Dumbo Jul 19 '16 at 17:21
  • @SaeidYazdani it's all the way you list the images in the list – Shai Jul 19 '16 at 17:27
  • @Shai: Regarding what you mentioned that "Therefore, you need two different files /path/to/labels/train.txt and /path/to/labels/test.txt the image names in those files should be different, but they can point to images in the same or in different folders - it's up t o you to organize them.", is it OK if I use the same filename but it's put in different directories i.e. in train and val directories, just like I did in (http://stackoverflow.com/questions/42559763/directory-structure-and-labeling-in-caffe)? – alfa_80 Mar 02 '17 at 17:29
  • 1
    @alfa_80 if the paths (directories) are different then the files are **not** the same. It's up to you to organize the images. Just make sure you do not test on images that were in the training set -this is cheating – Shai Mar 02 '17 at 18:49
  • @Shai: Thanks for the answer. Yes, they both are totally different images although having the same name, so as to avoid over-fitting (remembering instead of having generalization capability). – alfa_80 Mar 02 '17 at 20:42
  • @Shai I can't create the binary. When I run `make tools` I get `make: Nothing to be done for 'tools'.`. What is happening? – fabda01 Sep 06 '18 at 15:25
  • @yellow01 this usually means your tools are already compiled and up to date – Shai Sep 06 '18 at 15:50