how to serialize / deserialize List with protobuf?

Question

I want to serialize List<ArchiveData> but it nearly always fail. Protobuff throw following exception:

Invalid wire-type; this usually means you have over-written a file without truncating or setting the length; see Using Protobuf-net, I suddenly got an exception about an unknown wire-type

I have read the post but still can not find a solution. How can i serialize and deserialize this?

My Class and Struct:

[Serializable, ProtoContract(Name = @"Archive"), ProtoInclude(1, typeof(List<ArchiveData>))]
public partial class Archive : IExtensible
{
    [ProtoMember(1, IsRequired = true, OverwriteList = true, Name = @"data", DataFormat = DataFormat.Default)]
    public List<ArchiveData> data { get; set; }

    public Archive()
    {
        data = new List<ArchiveForm.ArchiveData>();
    }

    private IExtension extensionObject;
    IExtension IExtensible.GetExtensionObject(bool createIfMissing)
    {
        return Extensible.GetExtensionObject(ref extensionObject, createIfMissing);
    }
}

public struct ArchiveData
{
    [ProtoMember(1, IsRequired = false, OverwriteList = true, Name = @"sourcefolder", DataFormat = DataFormat.Default)]
    [global::System.ComponentModel.DefaultValue("")]
    public string sourcefolder { get; set; }

    [ProtoMember(2, IsRequired = false, OverwriteList = true, Name = @"destinationfolder", DataFormat = DataFormat.Default)]
    [global::System.ComponentModel.DefaultValue("")]
    public string destinationfolder { get; set; }

    [ProtoMember(3, IsRequired = false, OverwriteList = true, Name = @"period", DataFormat = DataFormat.FixedSize)]
    [global::System.ComponentModel.DefaultValue((int)0)]
    public int period { get; set; }

    public ArchiveData(string sfolder = "", string dfolder = "", int priod = 0)
    {
        sourcefolder = sfolder;
        destinationfolder = dfolder;
        period = priod;
    }
}

I serialize that with following method:

public static void Refresh(ref Archive arc)
{
    if (File.Exists(probuffile))
    {
        using (var fs = File.OpenRead(probuffile))
        {
            arc = Serializer.Deserialize<Archive>(fs);
        }
    }
}

And i deserialize that with following method:

public static void Update(Archive arc)
{
    using (var fs = File.Create(probuffile))
    {
        Serializer.Serialize<Archive>(fs, arc);
        fs.SetLength(fs.Position);
    }
}

And i use it:

Archive archive = new Archive();
//Add some ArchiveData.
Refresh(ref archive);

------------------------------Edit------------------------------

This section has been added for more information. When I use SerializeWithLengthPrefix / DeserializeWithLengthPrefix function like following code, it work properly every time for the first class that i use the Deserialize function. But it return null for the second class that i use the Deserialize function.

[Serializable, ProtoContract(Name = @"OptionData")]
public class OptionData : IExtensible
{
    [ProtoMember(1, IsRequired = false, OverwriteList = true, Name = @"StartWithWindows", DataFormat = DataFormat.Default)]
    [DefaultValue(false)]
    public bool StartWithWindows { get; set; }

    [ProtoMember(2, IsRequired = false, OverwriteList = true, Name = @"AutoBackup", DataFormat = DataFormat.Default)]
    [DefaultValue(false)]
    public bool AutoBackup { get; set; }

    [ProtoMember(3, IsRequired = false, OverwriteList = true, Name = @"Speed", DataFormat = DataFormat.FixedSize)]
    [DefaultValue((int)0)]
    public int Speed { get; set; }

    private IExtension extensionObject;
    IExtension IExtensible.GetExtensionObject(bool createIfMissing)
    {
        return Extensible.GetExtensionObject(ref extensionObject, createIfMissing);
    }
}

public static void Update(OptionData op)
{
    using (var fs = File.Create(probuffile))
    {
        Serializer.SerializeWithLengthPrefix(fs, op, PrefixStyle.Base128, 3);
    }
}

public static void Update(Archive arc)
{
    using (var fs = File.Create(probuffile))
    {
        Serializer.SerializeWithLengthPrefix<Archive>(fs, arc, PrefixStyle.Base128, 2);
    }
}

public static void Refresh(ref OptionData op)
{
    if (File.Exists(probuffile))
    {
        using (var fs = File.OpenRead(probuffile))
        {
            op = Serializer.DeserializeWithLengthPrefix<OptionData>(fs, PrefixStyle.Base128, 3);
        }
    }
}

public static void Refresh(ref Archive arc)
{
    if (File.Exists(probuffile))
    {
        using (var fs = File.OpenRead(probuffile))
        {
            arc = Serializer.DeserializeWithLengthPrefix<Archive>(fs, PrefixStyle.Base128, 2);
        }
    }
}

Re the edit: are they in the same file? or different files? if you want to read two successive things from the same file, you need to read them successively on a single `File.OpenRead` session. Otherwise, every time you call `File.OpenRead` it will be back at the start of the file and will be reading the wrong data. You don't show how `Update` and `Refresh` are used, so I can't be 100% sure. Again, the best thing to do - like in my answer - is to post some **fully runnable** code (with a `Main()` method etc) that demonstrates the problem actually happening. — Marc Gravell, Mar 31 '17 at 08:18
I give up to try serialize/deserialize two diffrent classes. I gather all in one and i works fine. If it slows down when the file size grows, i will try separate the class to two different classes. Thanks for answers :) — Burak Kocaman, Mar 31 '17 at 17:16

Marc Gravell · Answer 1 · 2017-03-29T08:27:47.773

I'd be happy to help, but I'm struggling to get it to fail. You say "but it nearly always fail", but the following runnable console exer works fine and writes a large range of data of various sizes:

using ProtoBuf;
using System;
using System.Collections.Generic;
using System.IO;
static class P
{
    static void Main()
    {
        var random = new Random(12345);
        // start with 100
        var archive = CreateData(random, 100);
        Update(archive);
        Refresh(ref archive);

        // make it bigger
        archive = CreateData(random, 200);
        Update(archive);
        Refresh(ref archive);

        // make it smaller
        archive = CreateData(random, 50);
        Update(archive);
        Refresh(ref archive);

        // go wild
        for (int i = 0; i < 1000; i++)
        {
            archive = CreateData(random, random.Next(0, 500));
            Update(archive);
            Refresh(ref archive);
        }

    }
    static Archive CreateData(Random random, int dataCount)
    {
        var archive = new Archive();
        var data = archive.data;
        for (int i = 0; i < dataCount; i++)
        {
            data.Add(new ArchiveData
            {
                period = random.Next(10000),
                sourcefolder = CreateString(random, 50),
                destinationfolder = CreateString(random, 50)
            });
        }
        return archive;
    }

    private static unsafe string CreateString(Random random, int maxLength)
    {
        const string Alphabet = "abcdefghijklmnopqrstuvwxyz0123456789";
        int len = random.Next(maxLength + 1);
        char* chars = stackalloc char[len];
        for (int i = 0; i < len; i++)
        {
            chars[i] = Alphabet[random.Next(Alphabet.Length)];
        }
        return new string(chars, 0, len);
    }

    static string probuffile = "my.bin";
    public static void Refresh(ref Archive arc)
    {
        if (File.Exists(probuffile))
        {
            using (var fs = File.OpenRead(probuffile))
            {
                arc = Serializer.Deserialize<Archive>(fs);
                Console.WriteLine("Read: {0} items, {1} bytes", arc.data.Count, fs.Length);
            }
        }
    }

    public static void Update(Archive arc)
    {
        using (var fs = File.Create(probuffile))
        {
            Serializer.Serialize<Archive>(fs, arc);
            fs.SetLength(fs.Position);
            Console.WriteLine("Wrote: {0} items, {1} bytes", arc.data.Count, fs.Length);
        }
    }
}

[Serializable, ProtoContract(Name = @"Archive"), ProtoInclude(1, typeof(List<ArchiveData>))]
public partial class Archive : IExtensible
{
    [ProtoMember(1, IsRequired = true, OverwriteList = true, Name = @"data", DataFormat = DataFormat.Default)]
    public List<ArchiveData> data { get; set; }

    public Archive()
    {
        data = new List<ArchiveData>();
    }

    private IExtension extensionObject;
    IExtension IExtensible.GetExtensionObject(bool createIfMissing)
    {
        return Extensible.GetExtensionObject(ref extensionObject, createIfMissing);
    }
}
[ProtoContract]
public struct ArchiveData
{
    [ProtoMember(1, IsRequired = false, OverwriteList = true, Name = @"sourcefolder", DataFormat = DataFormat.Default)]
    [global::System.ComponentModel.DefaultValue("")]
    public string sourcefolder { get; set; }

    [ProtoMember(2, IsRequired = false, OverwriteList = true, Name = @"destinationfolder", DataFormat = DataFormat.Default)]
    [global::System.ComponentModel.DefaultValue("")]
    public string destinationfolder { get; set; }

    [ProtoMember(3, IsRequired = false, OverwriteList = true, Name = @"period", DataFormat = DataFormat.FixedSize)]
    [global::System.ComponentModel.DefaultValue((int)0)]
    public int period { get; set; }

    public ArchiveData(string sfolder = "", string dfolder = "", int priod = 0)
    {
        sourcefolder = sfolder;
        destinationfolder = dfolder;
        period = priod;
    }
}

I'm guessing that whatever the difference is between your real code and the example above : is also the cause of the problem. If you help find what is different, I'd be more than happy to help however I reasonably can.

As a minor note: you don't actually need to set the length when using Create, since that is explicitly "throw away any existing data". The main time you need to worry about the length is when people use OpenWrite to write over an existing file, and then write less data than was already there. But there isn't a problem with explicitly setting the length after Create

I finally realise when fails.(I have another serializable class called OptionData). 1-)When i serialize/deserialize just OptionData, It works properly. 2-)When i serialize/deserialize just Archive, It works properly. But 3-)when i deserialize Archive while both OptionData and Archive was serialized, Archive is fails and OptionData works properly. — Burak Kocaman, Mar 29 '17 at 10:35
@BurakKocaman if you want me to look into that, I'm going to need some code. As a guess: are you serializing them back-to-back in the same stream (i.e. two calls to `Serialize`)? If so, you'll need to use the `*WithLengthPrefix` methods. However, a simpler fix for that would be to create a single root object that has the two items as members 1/2, i.e. `[ProtoContract] class MyRoot { [ProtoMember(1)] public Archive Archive {get;set;} [ProtoMember(2)] public OptionData Options {get;set;} }` — Marc Gravell, Mar 29 '17 at 11:33
Thanks for answer. I tried to use WithLengthPrefix and I thought I should look for a different solution when it didn't work. But after your answer, i'm sure i used it wrong. I will try to use WithLengthPrefix until it work properly :D Thanks again. — Burak Kocaman, Mar 29 '17 at 13:30
@BurakKocaman if you want me to fix what you have, I can do that - but I'd need to see code that fails — Marc Gravell, Mar 29 '17 at 13:35

how to serialize / deserialize List with protobuf?

1 Answers1