0

In my game I used the attribute [Serializable] on relevant classes to serialise my campaign (a collection of maps and levels) into a single file but now the campaign is nearing 20MB in size and it's becoming too slow for serialising/deserialising.

It currently takes about 11 seconds to deserialise the campaign into memory from disk.

I know about protobuf-net but I want to try the ISerializable approach for now. Would implementing the ISerializable interface on a class interfere with the deserialisation of a previous version of that class which only used [Serializable] and [NonSerialized] attributes to serialise?

I imagine I could create an ISerializable copy of each relevant class and copy data from one to another but that sounds like a recipe for disaster.

Is it possible to tell the BinaryFormatter to ignore any ISerializable attributes and only deserialise using the [Serializable] and [NonSerialized] attributes?

Update 1: I use a custom binder to fix up type and assembly name differences but this made no noticable difference when not in use.

Update 2: SSD embedded content deserialisation takes ~12.2s, SSD disk file deserialisation takes ~11.4s, MemoryStream serialisation takes ~3.4s and MemoryStream deserialisation takes ~11.2s. I feel silly but I forgot to mention this is all done on a seperate thread. I'll increase it's priority and see if that changes anything.

Update 3: loading thread priority set to Highest trimmed off less than a second.

Thank you for your time.

MazK
  • 81
  • 6
  • Are those 20 MB uncompressed or compressed data? Just wondering if the disk I/O might the the primary cause of the delay and whether having compressed data might actually help. – Crusha K. Rool Aug 05 '17 at 10:18
  • These days 20MB files would most likely be read, cached, and placed wholly in memory in less than 1 second, even from an HDD. I suspect that the time taken is more to do with the deserialisation itself. – bazza Aug 05 '17 at 10:40
  • I agree with other responses. The time is probably the read/write time to the file and not the time to serialize. I would run an experiment and read file into a memory stream. Then serialize from the memory stream. You can add time to code to see how much time it takes to read into memory stream and time to serialize. – jdweng Aug 05 '17 at 11:03
  • @CrushaK.Rool It's uncompressed except for a simple compression technique on the tilegrids. It's running off an SSD. – MazK Aug 05 '17 at 11:45
  • @bazza That's what I thought. 20MB should be next to nothing. Would a deep data structure be the issus as it's somewhat dense. I save a file which is a MapSet class that contains Maps which contain LevelSets which contain Levels. They all have some other classes like MapNodes, EntityInfos etc.. – MazK Aug 05 '17 at 11:50
  • @jdweng Nice idea. I'll time that and compare to see if it's I/O related. It is running off an SSD but the test will be interesting nevertheless. – MazK Aug 05 '17 at 11:56
  • If disk I/O is taking a substantial percent of deserialization time, be aware that `BinaryFormatter` generates surprisingly large files (for a binary serializer). See https://maxondev.com/serialization-performance-comparison-c-net-formats-frameworks-xmldatacontractserializer-xmlserializer-binaryformatter-json-newtonsoft-servicestack-text/ and [Fastest way to serialize and deserialize .NET objects](https://stackoverflow.com/q/4143421). – dbc Aug 05 '17 at 17:43
  • 1
    I would be surprised if `ISerializable` sped things up much, since it basically replaces `BinaryFormatter` looking up property names in a dictionary with manual lookups in `SerializationInfo`. Rather than converting your entire code to use `ISerializable` you might prototype a test case with two classes that are identical other than that one uses `ISerializable`, then test the performance of serializing and deserializing lists of, say, 1,000,000 of each. – dbc Aug 05 '17 at 17:46
  • See also https://blogs.msdn.microsoft.com/youssefm/2009/07/10/comparing-the-performance-of-net-serializers/ – dbc Aug 05 '17 at 17:52
  • 1
    @dbc You're absolutely right! I implemented `ISerializable` on a few classes and they deserialised from their `[Serializable]` only versions with no issues. The filesize and subsequent deserialisation time however, increased noticably for even only ~100 levels. Thanks so much for those links. They seem to point to XML being the best overall for size and speed but I'm going to give protobuf-net a shot first because from what I've seen I won't be disappointed with the results. – MazK Aug 05 '17 at 21:16

2 Answers2

1

I ended up decorating a few classes with ISerializable and implementing the GetObjectData() methods and ISerializable constructors for them. I used the member variable names as the key for AddValue(string name, object value) and this deserialised my objects perfectly.

However, not only did the filesizes go up but the subsequent deserialisation time increased by ~30-40%.

I'm going to suggest that people use protobuf-net as it looks like the fastest way to serialise/deserialise data into a small file but XmlSerializer has also been shown to increase speed and decrease filesize substantially over BinaryFormatter

BIG UPDATE: Fully implemented protobuf-net for all the classes related to the main campaign content and oh boy was I not disappointed.

Total filesize for ~60 levels and all the world maps was just shy of 21MB and now using protobuf-net its 5.1MB!!!

Not only that but loading times were over 12 seconds for just loading the main campaign into memory. Now the game loads settings, user profiles, all textures and the main campaign in tens of milliseconds slower than a single second!!! I'm sorry but I just don't deserve such an astonishing improvement.

My game now has practically zero load time throughout thanks to Marc Gravell's excellent API

Thanks for everyones comments.

MazK
  • 81
  • 6
0

You can use Colfer to generate your serializable beans. No code changes needed; although using the native Marshal and unmarshal methods would be optimal. Even faster and smaller than ProtoBuf. ;-)

Pascal de Kloe
  • 523
  • 4
  • 12