Protobuf dictionary diserialization throws null key exception

Question

I'm working on project written on C# with protobuf-net. I have code which serializes/deserializes Dictionary<MyClass1, MyClass2> to file . Also, i have a file with serialized data. When i try to deserialize it, i receive an exception key can't be null. I don't understand how is it possible since Dictionary doesn't allows null, it looks like i couldn't serialize such dictionary. I think the file was corrupted, but i'm not sure. I tried to debug it, and looks like few keys and values were deserialized correctly, but in the middle of the deserialization process, null key occured and i see exception. I tried to use surrogate for ProductName as mentioned here but it doesn't help.

How to deserialize this file? May be there is a way to deserialize to some object instead of null for ProductName?

Exception:

System.ArgumentNullException: Value cannot be null. Parameter name: key at System.Collections.Generic.Dictionary2.Insert(TKey key, TValue value, Boolean add) at System.Collections.Generic.Dictionary2.System.Collections.Generic.ICollection>.Add(KeyValuePair`2 keyValuePair) at proto_10(Object , ProtoReader ) at ProtoBuf.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 704 at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, SerializationContext context) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 588 at ProtoBuf.Serializer.Deserialize[T](Stream source) in c:\Dev\protobuf-net\protobuf-net\Serializer.cs:line 77

Code:

[ DataContract ]
    public sealed class ProductName
    {
        [ DataMember( Order = 1 ) ]
        public string Name { get; private set; }

        public static readonly ProductName Undefined = Create( "Unknown" );

        private ProductName()
        {
        }

        private ProductName( string Name )
        {
            this.Name = Name.Trim();
        }

        public static ProductName Create( string Name )
        {
            Condition.Requires( Name, "Name" ).IsNotNullOrEmpty();

            return new ProductName( Name );
        }

        public static ProductName TryCreate( string Name )
        {
            return Name.IsValidName() ? new ProductName( Name ) : null;
        }

        public override string ToString()
        {
            return this.Name;
        }

        public override int GetHashCode()
        {
                var stableHashCodeIgnoringCase = this.Name.GetStableHashCodeIgnoringCase();
                return stableHashCodeIgnoringCase;
        }

        #region Equality members
        public bool Equals( ProductName other )
        {
            if( ReferenceEquals( null, other ) )
                return false;
            if( ReferenceEquals( this, other ) )
                return true;
            return string.Equals( this.Name, other.Name, StringComparison.InvariantCultureIgnoreCase );
        }

        public override bool Equals( object obj )
        {
            if( ReferenceEquals( null, obj ) )
                return false;
            if( ReferenceEquals( this, obj ) )
                return true;
            if( obj.GetType() != this.GetType() )
                return false;
            return this.Equals( ( ProductName )obj );
        }
        #endregion
    }


    [ DataContract ]
    public class ProductNameIndex
    {
        [ DataMember( Order = 1 ) ]
        public IDictionary< ProductName, ProductId > Products{ get; private set; }

        public ProductNameIndex()
        {
            this.Products = new Dictionary< ProductName, ProductId >();
        }       
    }

Answer 1

First, to test whether the file is in fact corrupt, you can try to follow the instructions in Recovering corrupted file serialize with Protobuf-net to check to see if the file is corrupt.

Next, regardless of whether the file is corrupt or not, to deserialize as far as possible, you can use Serializer.Merge<T>(Stream source, T instance) to merge the file onto a pre-allocated ProductNameIndex, like so:

var index = new ProductNameIndex();
try
{
    Serializer.Merge(stream, index);
}
catch (Exception ex)
{
    // Log the error
    Debug.WriteLine(ex);
}

index should now contain as many entries as it was possible to completely deserialize.

Now, if the file is corrupt, that is probably the best you can do unless you want to load it into a MemoryStream and attempt to manually fix it byte-by-byte. But if the file is not corrupt, you need to debug to find out what the problem is. One action to take is to compare the contract generated for ProductNameIndex with the contract actually used in the file. To see the contract for ProductNameIndex, you can do:

Debug.WriteLine(ProtoBuf.Meta.RuntimeTypeModel.Default.GetSchema(typeof(ProductNameIndex)));

Then compare the output from Recovering corrupted file serialize with Protobuf-net, which I believe only dumps top-level fields. If the field numbers and wire types do not match you will know your are reading a file whose root object is not actually ProductNameIndex. You can also use the contract infortmation from GetSchema() to manually step into repeated fields (strings) and nested objects.

Finally, if the contracts match but somehow the sending system is emitting a KeyValuePair_ProductName_ProductId message with a missing optional ProductName Key, you could modify your ProductNameIndex to ignore such invalid entries by using a surrogate array property:

[DataContract]
[ProtoContract]
public class ProductNameIndex
{
    [ProtoMember(1, Name = "Products")]
    KeyValuePair<ProductName, ProductId>[] ProductsSurrogateArray
    {
        get
        {
            return Products.ToArray();
        }
        set
        {
            if (value == null)
                return;
            foreach (var p in value)
            {
                if (p.Key == null)
                    Debug.WriteLine("Ignoring invalid null key");
                else
                    Products.Add(p);
            }
        }
    }

    [ProtoIgnore]
    [DataMember(Order = 1)]
    public IDictionary<ProductName, ProductId> Products { get; private set; }

    public ProductNameIndex()
    {
        this.Products = new Dictionary<ProductName, ProductId>();
    }
}

Or, if you cannot add protobuf attributes to your type, you could introduce a surrogate type for ProductNameIndex with the necessary checks, as explained here.

Finally, take a look at Using Protobuf-net, I suddenly got an exception about an unknown wire-type which has several useful suggestions for diagnosing protobuf-net problems, including:

The most likely cause (in my experience) is that you have overwritten an existing file, but have not truncated it; i.e. it was 200 bytes; you've re-written it, but with only 182 bytes. There are now 18 bytes of garbage on the end of your stream that is tripping it up. Files must be truncated when re-writing protocol buffers.