PyYAML path at serialization time vs deserialization time

Question

I am working on a game engine which includes a simple GUI development tool. The GUI tool allows a user to define various entities and components, which can then be saved in a configuration file. When the game engine runtime loads the configuration file, it can determine how to create the various entities and components for use in the game.

For a configuration file saving mechanism, I am using PyYAML. The issue that I am having stems from the fact that the serialization process occurs in a module which is in a different directory than the module which loads and parses the file through PyYAML.

Simplified Serializer

import yaml

def save_config(context, file_name):
    config_file = file(file_name, 'w')
    # do some various processing on the context dict object
    yaml.dump(context, config_file)
    config_file.close()

This takes the context object, which is a dict that represents various game objects, and writes it to a config file. This works without issue.

Simplified Deserializer in engine

import yaml

def load(file_name):
    config_file = open(file_name, 'r')
    context = yaml.load(config_file)
    return context

This is where the problem occurs. On yaml.load(config_file), I will receive an error, because it fails to find a various name on a certain module. I understand why this is happening. For example, when I serialize the config file, it will list an AssetComponent (a component type in the engine) as being at engine.common.AssetComponent. However, from the deserializer's perspective, the AssetComponent should just be at common.AssetComponent (because the deserialization code itself exists within the engine package), so it fails to find it under engine.

Is there a way to manually handle paths when serializing or deserializing with PyYAML? I would like to make sure they both happen from the same "perspective."

Edit: The following shows what a problematic config file might look like, followed by what the manually corrected config would look like

Problematic

!!python/object/apply:collections.defaultdict
args: [!!python/name:__builtin__.dict ''] 
dictitems:
  assets:
  - !!python/object:common.Component
    component: !!python/object:engine.common.AssetComponent {file_name: ../content/sticksheet.png,
      surface: null}
    text: ../content/sticksheet.png
    type_name: AssetComponent

Corrected

!!python/object/apply:collections.defaultdict
args: [!!python/name:__builtin__.dict ''] 
dictitems:
  assets:
  - !!python/object:tools.common.Component
    component: !!python/object:common.AssetComponent {file_name: ../content/sticksheet.png,
      surface: null}
    text: ../content/sticksheet.png
    type_name: AssetComponent

Can you add an example of a config file that displays the problem? — Hans Then, May 27 '14 at 06:24
Do the serialization and deserialization routines each have their own `__main__` routine? — Hans Then, May 27 '14 at 06:43
The serialization and deserialization routines do not each have their own `__main__` routines, however they are each called by separate modules which have their own `__main__` routines. — Haz, May 27 '14 at 14:24

score 0 · Answer 1 · answered May 22 '14 at 05:48

0

You can explicitly declare the Python type of an object in a YAML document:

!!python/object:module_foo.ClassFoo {
    attr_foo: "spam",
    …,
}

answered May 22 '14 at 05:48

bignose

30,281
14
77
110

I've been doing something similar in order to at least be able to continue working, by doing some strategic search/replace. The challenge is in getting this to happen automatically, for any arbitrary object which resides in either the `tools` package (where the serializer is) or the `engine` package (where the deserializer is). – Haz May 22 '14 at 17:25
Please update the question to show how you're already using this, and explain in the question why this documented feature isn't meeting your requirements. – bignose May 23 '14 at 12:53
I'm using it by doing search/replace on the saved YAML file after serializing the structure. This doesn't meet the requirements because I don't see anything in the documentation about how to (at runtime) change the path in the `!!python/object` tags. For example: currently it is serializing something at `!!python/object:common.Component` and I want it to instead save it as `!!python/object:tools.common.Component` so that it resolves properly when deserialized from an arbitrary location. – Haz May 23 '14 at 15:17

score 0 · Accepted Answer · edited May 23 '17 at 11:57

Your problem lies in a mismatch between your package structure and your __main__ routines. The module containing your __main__ will be inside a package, but it has no way of knowing that. So, therefore, you will use imports relative to the location of the file containing __main__ and not relative to the top level structure of your package.

See Relative imports for the billionth time for a longer (and probably better) explanation.

So, how can you fix it?

Inside the file containing __main__ you do:

from tools.common import Component
# instead of from common import Component

c = Component()
print yaml.dump(c)

Another thing you must ensure is that python will know how to load your modules. If you have installed your package this will be done automatically, but during development this is usually not the case. So during development you will also want to make your development modules findable.

The easiest way (but not very clean) is to use sys.path.append('the directory containing tools and engine'). Another way (cleaner) is to set the PYTHONPATH environment variable to include your top level directory containing tools and engine.

PyYAML path at serialization time vs deserialization time

2 Answers2