Excluding primary key in Django dumpdata with natural keys

Question

How do you exclude the primary key from the JSON produced by Django's dumpdata when natural keys are enabled?

I've constructed a record that I'd like to "export" so others can use it as a template, by loading it into a separate databases with the same schema without conflicting with other records in the same model.

As I understand Django's support for natural keys, this seems like what NKs were designed to do. My record has a unique name field, which is also used as the natural key.

So when I run:

from django.core import serializers
from myapp.models import MyModel
obj = MyModel.objects.get(id=123)
serializers.serialize('json', [obj], indent=4, use_natural_keys=True)

I would expect an output something like:

[
    {
        "model": "myapp.mymodel", 
        "fields": {
            "name": "foo", 
            "create_date": "2011-09-22 12:00:00", 
            "create_user": [
                "someusername"
            ]
        }
    }
]

which I could then load into another database, using loaddata, expecting it to be dynamically assigned a new primary key. Note, that my "create_user" field is a FK to Django's auth.User model, which supports natural keys, and it output as its natural key instead of the integer primary key.

However, what's generated is actually:

[
    {
        "pk": 123,
        "model": "myapp.mymodel", 
        "fields": {
            "name": "foo", 
            "create_date": "2011-09-22 12:00:00", 
            "create_user": [
                "someusername"
            ]
        }
    }
]

which will clearly conflict with and overwrite any existing record with primary key 123.

What's the best way to fix this? I don't want to retroactively change all the auto-generated primary key integer fields to whatever the equivalent natural keys are, since that would cause a performance hit as well as be labor intensive.

Edit: This seems to be a bug that was reported...2 years ago...and has largely been ignored...

score 12 · Answer 1 · edited Jan 12 '20 at 09:31

Updating the answer for anyone coming across this in 2018 and beyond.

There is a way to omit the primary key through the use of natural keys and unique_together method. Taken from the Django documentation on serialization:

You can use this command to test :

python manage.py dumpdata app.model --pks 1,2,3 --indent 4 --natural-primary --natural-foreign > dumpdata.json ;

Serialization of natural keys

So how do you get Django to emit a natural key when serializing an object? Firstly, you need to add another method – this time to the model itself:

class Person(models.Model):
    objects = PersonManager()

    first_name = models.CharField(max_length=100)
    last_name = models.CharField(max_length=100)
    birthdate = models.DateField()

    def natural_key(self):
        return (self.first_name, self.last_name)

    class Meta:
        unique_together = (('first_name', 'last_name'),)

That method should always return a natural key tuple – in this example, (first name, last name). Then, when you call serializers.serialize(), you provide use_natural_foreign_keys=True or use_natural_primary_keys=True arguments:

serializers.serialize('json', [book1, book2], indent=2, use_natural_foreign_keys=True, use_natural_primary_keys=True)

When use_natural_foreign_keys=True is specified, Django will use the natural_key() method to serialize any foreign key reference to objects of the type that defines the method.

When use_natural_primary_keys=True is specified, Django will not provide the primary key in the serialized data of this object since it can be calculated during deserialization:

    {
        "model": "store.person",
        "fields": {
            "first_name": "Douglas",
            "last_name": "Adams",
            "birth_date": "1952-03-11",
        }
    }

score 9 · Accepted Answer · edited May 23 '17 at 12:14

The problem with json is that you can't omit the pk field since it will be required upon loading of the fixture data again. If not existing, json will fail with

$ python manage.py loaddata some_data.json
[...]
File ".../django/core/serializers/python.py", line 85, in Deserializer
data = {Model._meta.pk.attname : Model._meta.pk.to_python(d["pk"])}
KeyError: 'pk'

As pointed out in the answer to this question, you can use yaml or xml if you really want to omit the pk attribute OR just replace the primary key value with null.

import re
from django.core import serializers

some_objects = MyClass.objects.all()
s = serializers.serialize('json', some_objects, use_natural_keys=True)
# Replace id values with null - adjust the regex to your needs
s = re.sub('"pk": [0-9]{1,5}', '"pk": null', s)

This is exactly the solution I discovered myself. – Cerin Mar 15 '12 at 12:48 — Cerin, Mar 15 '12 at 12:48
It seems you can omit the `pk` field even in json now. – Aug 18 '16 at 16:03 — , Aug 18 '16 at 16:03

Mariusz Jamro · Answer 3 · 2012-02-27T14:24:04.040

4

Override the Serializer class in a separate module:

from django.core.serializers.json import Serializer as JsonSerializer

class Serializer(JsonSerializer):

    def end_object(self, obj):
        self.objects.append({
            "model"  : smart_unicode(obj._meta),
            "fields" : self._current, 
            # Original method adds the pk here
        })
        self._current = None

Register it in Django:

serializers.register_serializer("json_no_pk", "path.to.module.with.custom.serializer")

Add use it:

serializers.serialize('json_no_pk', [obj], indent=4, use_natural_keys=True)

edited Feb 27 '12 at 14:24

answered Feb 27 '12 at 14:16

Mariusz Jamro

30,615
24
120
162

2

I'd prefer this solution, however deserialization fails without pk value. You should set ``"pk": None,`` rather then deleting the whole field – Igor Nov 15 '13 at 08:53

Excluding primary key in Django dumpdata with natural keys

3 Answers3

Linked