Parsing a binary file. What is a modern way?

Question

I have a binary file with some layout I know. For example let format be like this:

• 2 bytes (unsigned short) - length of a string
• 5 bytes (5 x chars) - the string - some id name
• 4 bytes (unsigned int) - a stride
• 24 bytes (6 x float - 2 strides of 3 floats each) - float data

The file should look like (I added spaces for readability):

5 hello 3 0.0 0.1 0.2 -0.3 -0.4 -0.5

Here 5 - is 2 bytes: 0x05 0x00. "hello" - 5 bytes and so on.

Now I want to read this file. Currently I do it so:

• load file to ifstream
• read this stream to char buffer[2]
• cast it to unsigned short: unsigned short len{ *((unsigned short*)buffer) };. Now I have length of a string.
• read a stream to vector<char> and create a std::string from this vector. Now I have string id.
• the same way read next 4 bytes and cast them to unsigned int. Now I have a stride.
• while not end of file read floats the same way - create a char bufferFloat[4] and cast *((float*)bufferFloat) for every float.

This works, but for me it looks ugly. Can I read directly to unsigned short or float or string etc. without char [x] creating? If no, what is the way to cast correctly (I read that style I'm using - is an old style)?

P.S.: while I wrote a question, the more clearer explanation raised in my head - how to cast arbitrary number of bytes from arbitrary position in char [x]?

Update: I forgot to mention explicitly that string and float data length is not known at compile time and is variable.

Answer 1

If it is not for learning purpose, and if you have freedom in choosing the binary format you'd better consider using something like protobuf which will handle the serialization for you and allow to interoperate with other platforms and languages.

If you cannot use a third party API, you may look at QDataStream for inspiration

• Documentation
• Source code

Answer 2

The C way, which would work fine in C++, would be to declare a struct:

#pragma pack(1)

struct contents {
   // data members;
};

Note that

• You need to use a pragma to make the compiler align the data as-it-looks in the struct;
• This technique only works with POD types

And then cast the read buffer directly into the struct type:

std::vector<char> buf(sizeof(contents));
file.read(buf.data(), buf.size());
contents *stuff = reinterpret_cast<contents *>(buf.data());

Now if your data's size is variable, you can separate in several chunks. To read a single binary object from the buffer, a reader function comes handy:

template<typename T>
const char *read_object(const char *buffer, T& target) {
    target = *reinterpret_cast<const T*>(buffer);
    return buffer + sizeof(T);
}

The main advantage is that such a reader can be specialized for more advanced c++ objects:

template<typename CT>
const char *read_object(const char *buffer, std::vector<CT>& target) {
    size_t size = target.size();
    CT const *buf_start = reinterpret_cast<const CT*>(buffer);
    std::copy(buf_start, buf_start + size, target.begin());
    return buffer + size * sizeof(CT);
}

And now in your main parser:

int n_floats;
iter = read_object(iter, n_floats);
std::vector<float> my_floats(n_floats);
iter = read_object(iter, my_floats);

Note: As Tony D observed, even if you can get the alignment right via #pragma directives and manual padding (if needed), you may still encounter incompatibility with your processor's alignment, in the form of (best case) performance issues or (worst case) trap signals. This method is probably interesting only if you have control over the file's format.

Answer 3

Currently I do it so:

• load file to ifstream

• read this stream to char buffer[2]

• cast it to unsigned short: unsigned short len{ *((unsigned short*)buffer) };. Now I have length of a string.

That last risks a SIGBUS (if your character array happens to start at an odd address and your CPU can only read 16-bit values that are aligned at an even address), performance (some CPUs will read misaligned values but slower; others like modern x86s are fine and fast) and/or endianness issues. I'd suggest reading the two characters then you can say (x[0] << 8) | x[1] or vice versa, using htons if needing to correct for endianness.

• read a stream to vector<char> and create a std::string from this vector. Now I have string id.

No need... just read directly into the string:

std::string s(the_size, ' ');

if (input_fstream.read(&s[0], s.size()) &&
    input_stream.gcount() == s.size())
    ...use s...

• the same way read next 4 bytes and cast them to unsigned int. Now I have a stride. while not end of file read floats the same way - create a char bufferFloat[4] and cast *((float*)bufferFloat) for every float.

Better to read the data directly over the unsigned ints and floats, as that way the compiler will ensure correct alignment.

This works, but for me it looks ugly. Can I read directly to unsigned short or float or string etc. without char [x] creating? If no, what is the way to cast correctly (I read that style I'm using - is an old style)?

struct Data
{
    uint32_t x;
    float y[6];
};
Data data;
if (input_stream.read((char*)&data, sizeof data) &&
    input_stream.gcount() == sizeof data)
    ...use x and y...

Note the code above avoids reading data into potentially unaligned character arrays, wherein it's unsafe to reinterpret_cast data in a potentially unaligned char array (including inside a std::string) due to alignment issues. Again, you may need some post-read conversion with htonl if there's a chance the file content differs in endianness. If there's an unknown number of floats, you'll need to calculate and allocate sufficient storage with alignment of at least 4 bytes, then aim a Data* at it... it's legal to index past the declared array size of y as long as the memory content at the accessed addresses was part of the allocation and holds a valid float representation read in from the stream. Simpler - but with an additional read so possibly slower - read the uint32_t first then new float[n] and do a further read into there....

Practically, this type of approach can work and a lot of low level and C code does exactly this. "Cleaner" high-level libraries that might help you read the file must ultimately be doing something similar internally....

Answer 4

I actually implemented a quick and dirty binary format parser to read .zip files (following Wikipedia's format description) just last month, and being modern I decided to use C++ templates.

On some specific platforms, a packed struct could work, however there are things it does not handle well... such as fields of variable length. With templates, however, there is no such issue: you can get arbitrarily complex structures (and return types).

A .zip archive is relatively simple, fortunately, so I implemented something simple. Off the top of my head:

using Buffer = std::pair<unsigned char const*, size_t>;

template <typename OffsetReader>
class UInt16LEReader: private OffsetReader {
public:
    UInt16LEReader() {}
    explicit UInt16LEReader(OffsetReader const or): OffsetReader(or) {}

    uint16_t read(Buffer const& buffer) const {
        OffsetReader const& or = *this;

        size_t const offset = or.read(buffer);
        assert(offset <= buffer.second && "Incorrect offset");
        assert(offset + 2 <= buffer.second && "Too short buffer");

        unsigned char const* begin = buffer.first + offset;

        // http://commandcenter.blogspot.fr/2012/04/byte-order-fallacy.html
        return (uint16_t(begin[0]) << 0)
             + (uint16_t(begin[1]) << 8);
    }
}; // class UInt16LEReader

// Declined for UInt[8|16|32][LE|BE]...

Of course, the basic OffsetReader actually has a constant result:

template <size_t O>
class FixedOffsetReader {
public:
    size_t read(Buffer const&) const { return O; }
}; // class FixedOffsetReader

and since we are talking templates, you can switch the types at leisure (you could implement a proxy reader which delegates all reads to a shared_ptr which memoizes them).

What is interesting, though, is the end-result:

// http://en.wikipedia.org/wiki/Zip_%28file_format%29#File_headers
class LocalFileHeader {
public:
    template <size_t O>
    using UInt32 = UInt32LEReader<FixedOffsetReader<O>>;
    template <size_t O>
    using UInt16 = UInt16LEReader<FixedOffsetReader<O>>;

    UInt32< 0> signature;
    UInt16< 4> versionNeededToExtract;
    UInt16< 6> generalPurposeBitFlag;
    UInt16< 8> compressionMethod;
    UInt16<10> fileLastModificationTime;
    UInt16<12> fileLastModificationDate;
    UInt32<14> crc32;
    UInt32<18> compressedSize;
    UInt32<22> uncompressedSize;

    using FileNameLength = UInt16<26>;
    using ExtraFieldLength = UInt16<28>;

    using FileName = StringReader<FixedOffsetReader<30>, FileNameLength>;

    using ExtraField = StringReader<
        CombinedAdd<FixedOffsetReader<30>, FileNameLength>,
        ExtraFieldLength
    >;

    FileName filename;
    ExtraField extraField;
}; // class LocalFileHeader

This is rather simplistic, obviously, but incredibly flexible at the same time.

An obvious axis of improvement would be to improve chaining since here there is a risk of accidental overlaps. My archive reading code worked the first time I tried it though, which was evidence enough for me that this code was sufficient for the task at hand.

Answer 5

I had to solve this problem once. The data files were packed FORTRAN output. Alignments were all wrong. I succeeded with preprocessor tricks that did automatically what you are doing manually: unpack the raw data from a byte buffer to a struct. The idea is to describe the data in an include file:

BEGIN_STRUCT(foo)
    UNSIGNED_SHORT(length)
    STRING_FIELD(length, label)
    UNSIGNED_INT(stride)
    FLOAT_ARRAY(3 * stride)
END_STRUCT(foo)

Now you can define these macros to generate the code you need, say the struct declaration, include the above, undef and define the macros again to generate unpacking functions, followed by another include, etc.

NB I first saw this technique used in gcc for abstract syntax tree-related code generation.

If CPP is not powerful enough (or such preprocessor abuse is not for you), substitute a small lex/yacc program (or pick your favorite tool).

It's amazing to me how often it pays to think in terms of generating code rather than writing it by hand, at least in low level foundation code like this.

Answer 6

You should better declare a structure (with 1-byte padding - how - depends on compiler). Write using that structure, and read using same structure. Put only POD in structure, and hence no std::string etc. Use this structure only for file I/O, or other inter-process communication - use normal struct or class to hold it for further use in C++ program.

Answer 7

Since all of your data is variable, you can read the two blocks separately and still use casting:

struct id_contents
{
    uint16_t len;
    char id[];
} __attribute__((packed)); // assuming gcc, ymmv

struct data_contents
{
    uint32_t stride;
    float data[];
} __attribute__((packed)); // assuming gcc, ymmv

class my_row
{
    const id_contents* id_;
    const data_contents* data_;
    size_t len;

public:
    my_row(const char* buffer) {
        id_= reinterpret_cast<const id_contents*>(buffer);
        size_ = sizeof(*id_) + id_->len;
        data_ = reinterpret_cast<const data_contents*>(buffer + size_);
        size_ += sizeof(*data_) + 
            data_->stride * sizeof(float); // or however many, 3*float?

    }

    size_t size() const { return size_; }
};

That way you can use Mr. kbok's answer to parse correctly:

const char* buffer = getPointerToDataSomehow();

my_row data1(buffer);
buffer += data1.size();

my_row data2(buffer);
buffer += data2.size();

// etc.

Answer 8

I personally do it this way:

// some code which loads the file in memory
#pragma pack(push, 1)
struct someFile { int a, b, c; char d[0xEF]; };
#pragma pack(pop)

someFile* f = (someFile*) (file_in_memory);
int filePropertyA = f->a;

Very effective way for fixed-size structs at the start of the file.

Answer 9

Use a serialization library. Here are a few:

• Boost serialization and Boost fusion
• Cereal (my own library)
• Another library called cereal (same name as mine but mine predates theirs)
• Cap'n Proto

Answer 10

The Kaitai Struct library provides a very effective declarative approach, which has the added bonus of working across programming languages.

After installing the compiler, you will want to create a .ksy file that describes the layout of your binary file. For your case, it would look something like this:

# my_type.ksy

meta:
  id: my_type
  endian: be # for big-endian, or "le" for little-endian

seq: # describes the actual sequence of data one-by-one
  - id: len
    type: u2 # unsigned short in C++, two bytes
  - id: my_string
    type: str
    size: 5
    encoding: UTF-8
  - id: stride
    type: u4 # unsigned int in C++, four bytes
  - id: float_data
    type: f4 # a four-byte floating point number
    repeat: expr
    repeat-expr: 6 # repeat six times

You can then compile the .ksy file using the kaitai struct compiler ksc:

# wherever the compiler is installed
# -t specifies the target language, in this case C++
/usr/local/bin/kaitai-struct-compiler my_type.ksy -t cpp_stl

This will create a my_type.cpp file as well as a my_type.h file, which you can then include in your C++ code:

#include <fstream>
#include <kaitai/kaitaistream.h>
#include "my_type.h"

int main()
{
  std::ifstream ifs("my_data.bin", std::ifstream::binary);
  kaitai::kstream ks(&ifs);
  my_type_t obj(&ks);

  std::cout << obj.len() << '\n'; // you can now access properties of the object

  return 0;
}

Hope this helped! You can find the full documentation for Kaitai Struct here. It has a load of other features and is a fantastic resource for binary parsing in general.

Answer 11

I use ragel tool to generate pure C procedural source code (no tables) for microcontrollers with 1-2K of RAM. It did not use any file io, buffering, and produces both easy to debug code and .dot/.pdf file with state machine diagram.

ragel can also output go, Java,.. code for parsing, but I did not use these features.

The key feature of ragel is the ability to parse any byte-build data, but you can't dig into bit fields. Other problem is ragel able to parse regular structures but has no recursion and syntax grammar parsing.