Django: loaddata in migrations errors

Question

Something really annoying is happening to me since using Django migrations (not south) and using loaddata for fixtures inside of them.

Here is a simple way to reproduce my problem:

create a new model Testmodel with 1 field field1 (CharField or whatever)
create an associated migration (let's say 0001) with makemigrations
run the migration
and add some data in the new table
dump the data in a fixture testmodel.json
create a migration with call_command('loaddata', 'testmodel.json'): migration 0002
add some a new field to the model: field2
create an associated migration (0003)

Now, commit that, and put your db in the state just before the changes: ./manage.py migrate myapp zero. So you are in the same state as your teammate that didn't get your changes yet.

If you try to run ./manage.py migrate again you will get a ProgrammingError at migration 0002 saying that "column field2 does not exist".

It seems it's because loaddata is looking into your model (which is already having field2), and not just applying the fixture to the db.

This can happen in multiple cases when working in a team, and also making the test runner fail.

Did I get something wrong? Is it a bug? What should be done is those cases?

--

I am using django 1.7

did you were applying loading of data inside the RunPython's forward func? — andilabs, Dec 18 '18 at 19:43

score 17 · Accepted Answer · edited Aug 23 '16 at 22:58

loaddata command will simply call serializers. Serializers will work on models state from your models.py file, not from current migration, but there is little trick to fool default serializer.

First, you don't want to use that serializer by call_command but rather directly:

from django.core import serializers

def load_fixture(apps, schema_editor):
    fixture_file = '/full/path/to/testmodel.json'
    fixture = open(fixture_file)
    objects = serializers.deserialize('json', fixture, ignorenonexistent=True)
    for obj in objects:
        obj.save()
    fixture.close()

Second, monkey-patch apps registry used by serializers:

from django.core import serializers

def load_fixture(apps, schema_editor):
    original_apps = serializers.python.apps
    serializers.python.apps = apps
    fixture_file = '/full/path/to/testmodel.json'
    fixture = open(fixture_file)
    objects = serializers.deserialize('json', fixture, ignorenonexistent=True)
    for obj in objects:
        obj.save()
    fixture.close()
    serializers.python.apps = original_apps

Now serializer will use models state from apps instead of default one and whole migration process will succeed.

I'm doing this now, and it seems like it's not incrementing the primary ID counter in postgres. My tests create some more users than the ones that are in my fixtures and I get 2 (the number of fixture users) duplicate ID errors before the tests start passing. — Aaron McMillin, Jun 02 '17 at 20:13

score 4 · Answer 2 · answered Jun 02 '17 at 21:16

To expand on the answer from GwynBleidD and mix in this issue since Postgres won't reset the primary key sequences when loaded this way (https://stackoverflow.com/a/14589706/401636)

I think I now have a failsafe migration for loading fixture data.

utils.py:

import os

from io import StringIO

import django.apps

from django.conf import settings
from django.core import serializers
from django.core.management import call_command
from django.db import connection


os.environ['DJANGO_COLORS'] = 'nocolor'


def reset_sqlsequence(apps=None, schema_editor=None):
    """Suitable for use in migrations.RunPython"""

    commands = StringIO()
    cursor = connection.cursor()
    patched = False

    if apps:
        # Monkey patch django.apps
        original_apps = django.apps.apps
        django.apps.apps = apps
        patched = True
    else:
        # If not in a migration, use the normal apps registry
        apps = django.apps.apps

    for app in apps.get_app_configs():
        # Generate the sequence reset queries
        label = app.label
        if patched and app.models_module is None:
            # Defeat strange test in the mangement command
            app.models_module = True
        call_command('sqlsequencereset', label, stdout=commands)
        if patched and app.models_module is True:
            app.models_module = None

    if patched:
        # Cleanup monkey patch
        django.apps.apps = original_apps

    sql = commands.getvalue()
    print(sql)
    if sql:
        # avoid DB error if sql is empty
        cursor.execute(commands.getvalue())


class LoadFixtureData(object):
    def __init__(self, *files):
        self.files = files

    def __call__(self, apps=None, schema_editor=None):
        if apps:
            # If in a migration Monkey patch the app registry
            original_apps = serializers.python.apps
            serializers.python.apps = apps

        for fixture_file in self.files:
            with open(fixture_file) as fixture:
                objects = serializers.deserialize('json', fixture)

                for obj in objects:
                    obj.save()

        if apps:
            # Cleanup monkey patch
            serializers.python.apps = original_apps

And now my data migrations look like:

# -*- coding: utf-8 -*-
# Generated by Django 1.11.1 on foo
from __future__ import unicode_literals

import os

from django.conf import settings
from django.db import migrations

from .utils import LoadFixtureData, reset_sqlsequence


class Migration(migrations.Migration):

    dependencies = [
        ('app_name', '0002_auto_foo'),
    ]

    operations = [
        migrations.RunPython(
            code=LoadFixtureData(*[
                os.path.join(settings.BASE_DIR, 'app_name', 'fixtures', fixture) + ".json"
                for fixture in ('fixture_one', 'fixture_two',)
            ]),
            # Reverse will NOT remove the fixture data
            reverse_code=migrations.RunPython.noop,
        ),
        migrations.RunPython(
            code=reset_sqlsequence,
            reverse_code=migrations.RunPython.noop,
        ),
    ]

Gocht · Answer 3 · 2015-10-07T21:45:22.490

2

When you run python manage.py migrate it's trying to load your testmodel.json in fixtures folder, but your model (after updated) does not match with data in testmodel.json. You could try this:

Change your directory from fixture to _fixture.
Run python manage.py migrate
Optional, you now can change _fixture by fixture and load your data as before with migrate command or load data with python manage.py loaddata app/_fixtures/testmodel.json

edited Oct 07 '15 at 21:45

answered Oct 02 '15 at 16:59

Gocht

9,924
3
42
81

What you suggest would produce exactly the same error, but with a good reason in this case. NB: https://docs.djangoproject.com/en/1.7/howto/initial-data/#automatically-loading-initial-data-fixtures – lajarre Oct 02 '15 at 17:01
I've had this same problem, now I loaddata manually (test data is in _fixture folder) because in my dev stage my models are still changing. – Gocht Oct 02 '15 at 17:03
I get your point. But finally I'd like to be able to have my fixtures automatically loaded... – lajarre Oct 02 '15 at 17:05

Django: loaddata in migrations errors

3 Answers3

Linked