Serialize data from xml model into custom binary format

Question

I am looking here for advice on how to handle my problem, and to know if there is already some tools working like this, that I am not aware of... Well, I have to communicate a lot of data from my system to another using their custom binary file format to exchange data. I have many of these (tlv like) binary files to generate and I am searching how to achieve my goal simply without handwriting to much code because I was told that the files format could evolve pretty soon.

For example here are 2 tables I have in my system (a list of products and their labels in different iso code) :

Table of products :

----------
SAL_PRODUCT
----------
ID NUMBER(3)
VENDOR_ID NUMBER (3)
MODEL_REF VARCHAR2(30)
----------

Table of product labels :

----------
SAL_PRODUCT_LABEL
----------
PRODUCT_ID NUMBER(3)
ISO_LANG_CODE VARCHAR2(5)
LABEL VARCHAR2(40)
----------

I have to produce a binary file similar as this structure :

• [Byte-size] <-- total size in byte of the message

• [Elm-count] <-- number of products

  • [Byte-size] <-- size in byte of first message (product n° 1)

  • [ID] <-- 1 byte product ID

  • [VENDORID] <-- 1 byte product vendor ID

    • [Elm-count] <-- number of product labels

      • [Byte-size] <-- size in byte of first message (label n° 1 of product n° 1)

      • [LANG_ID] <-- string ISO lang code on 5 caracters length

      • [LABELID] <-- String Label of product n° 1 for the current lang_id on 30 caracters length

I figure out I must create a file generator able to work with an xml metadata file as this :

<messages type="product" select="select ID,VENDOR_ID, MODEL_REF from SAL_PRODUCT order by 1">

    <message>

        <prop type="ID"/>
        <prop type="VENDOR_ID" />

        <messages type="labels" select="select product_id, ISO_LANG_CODE as LANGID, LABEL from  SAL_PRODUCT_LABEL" joinclause="product.ID = labels.product_id " >

           <message  >  
            <prop type="LANGID" />
            <prop type="LABEL" />
           </message>


        </messages>

        <prop type="MODEL_REF"/>

    </message>  


</messages>     

But all my C# code sould be dynamic and able to parse and work with the nested structure.... From here I do not know how to handle my problem :

• does I have chose the right solution ?

• how would you handle this kind of problem?

Thank you for any help

Answer 1

One of the simplest way is to do it is to do it in the following two steps:

1. Create strongly typed model classes

Use strongly typed model classes. They can be either handwritten/generated classes to deserialize XML to, or, if you use some existing database storage, then Entity Framework database first approach can be used to create them from existing database tables.

What will it give to you? - You won't need any custom descriptors for your data (I mean those not so pretty looking XML format descriptors). The data models will define their format itself (plus, perhaps, some attributes) and the risk of possible mistakes due to mismanaged or not up to date serialization configuration will be greatly reduced.

Any disadvantages? - Well, it won't be as easy to version (I mean serialize or deserialize older files) - you will have to use one model for each version (like ModelV1, ModelV2...) and one serializer for each binary file format version (SerializerFormatV1, SerializerFormatV2...). But with dynamic structure it would have been even more difficult to accomplish.

2. Create custom serializer

As far as I know, .NET framework doesn't provide any standard serialization-deserialization infrastructure that you can easily integrate your custom serializer into.

Well, there is ISerializable interface and the entire infrastructure over it - https://msdn.microsoft.com/en-us/library/ty01x675(v=vs.110).aspx , but that's mostly .NET 2.0 time tech (not used in either DataContractSerializer or Protobuf/Newtonsoft.Json serialization frameworks). And you won't get anything by adhering to it.

So you should just implement custom serializer, that will reflect over your model types' properties and write the data according to that custom binary format.

How to enumerate all properties - Recursively Get Properties & Child Properties Of A Class

Example of a similar custom serializer (just for flat text file) - Custom serialization of an object in .NET

It will be something like:

public class OtherCompanyBinaryFormatSerializer
{
    public void Serialize<T>(Stream stream, T object); { ... }
    public T Deserialize<T>(Stream stream); { ... }
}

---

There are also some things you should be mindful about:

1. Nested structure - while enumerating your properties don't forget to properly recurs to serialize nested structures.
2. Cyclic reference - that's a bit more difficult and I can't find any good advises on how to do it. But if your data can't have such things, then you are probably safe.
3. Versioning - once upon a time the format changes, either of your own data, or of that binary file format. But you will still have to deal with older data. How to handle it? As I have already told in step 1 - with version tagged data and serializers. Some internet searches on "version tolerant serialization" will give you more ideas.