Mocking Django Storages Model ImageField backend S3

Question

I have a model with an ImageField that is backed by django-storages' S3Boto. I have a test the exercises the "upload image" view, but the fact that it is uploading the image to S3 is slowing down my test suite.

In the interest of speeding up my tests, what is the best practice for dealing with this issue? Should I mock out S3Boto? Perhaps there is a memory backed storage backend that works well for testing (automatic cleanup would be nice!)?

meshy · Accepted Answer · 2017-06-09T10:42:02.737

I just had this problem too. I got much faster tests by using dj-inmemorystorage.

The quick way of setting this up is by creating a test_settings.py in the same folder as your settings:

from settings import *
DEFAULT_FILE_STORAGE = 'inmemorystorage.InMemoryStorage'

...and calling ./manage.py test --settings=project.test_settings to run the tests.

My preferred way is to set up a custom test runner:

In project/test_runner.py:

from django.conf import settings
from django.test.runner import DiscoverRunner

class FastTestRunner(DiscoverRunner):
    def setup_test_environment(self):
        super(FastTestRunner, self).setup_test_environment()
        # Don't write files
        settings.DEFAULT_FILE_STORAGE = 'inmemorystorage.InMemoryStorage'
        # Bonus: Use a faster password hasher. This REALLY helps.
        settings.PASSWORD_HASHERS = (
            'django.contrib.auth.hashers.MD5PasswordHasher',
        )

Note: This also sets the PASSWORD_HASHER, because it significantly improves User creation time. This should NOT be set in production.

In project/settings.py:

TEST_RUNNER = 'project.test_runner.FastTestRunner'

The requirements:

pip install dj-inmemorystorage

UPDATE: changed from django-inmemorystorage to dj-inmemorystorage.

UPDATE 2: Removed django-discover-runner, as it's now the default test runner in django, and fixed the link to the PASSWORD_HASHER related blog post.

score 5 · Answer 2 · answered Sep 08 '19 at 21:42

I also use S3Boto but for testing, I prefer having custom settings which include using the file system storage. You can have your custom settings declared in a file which you can then import and use in your test cases. Even so, you can mock the file storage so that the files are not actually written to disk.

Here's a sample test_settings.py

# myproject/myproject/test_settings.py

from django.test import override_settings

common_settings = override_settings(
    DEFAULT_FILE_STORAGE='django.core.files.storage.FileSystemStorage',
    PASSWORD_HASHERS=(
        'django.contrib.auth.hashers.UnsaltedMD5PasswordHasher',
    ),
)

Usage:


from django.test import TestCase

from myproject.test_settings import common_settings

@common_settings
class MyTestCase(TestCase):
    """Tests go here"""

On mocking the file system storage you can check out my answer here on SO.

A very helpful example, thank you!, On my case I have also overridden the parameter `MEDIA_ROOT` setting a path like this : `os.path.join(os.path.dirname(settings.BASE_DIR), 'media_test')` ( previously importing : `django.conf import settings` ) , based on the required fields for that specific storage configuration : https://docs.djangoproject.com/en/3.1/ref/files/storage/#django.core.files.storage.FileSystemStorage.location — Manuel Lazo, Mar 24 '21 at 19:09

yarbelk · Answer 3 · 2017-06-09T09:11:35.897

Just ran into this as well so I thought I'd put my solution up. My solution uses Mock

import mock
from django.core.files.storage import FileSystemStorage
from django.test import TestCase

class ATestCase(TestCase):
    def setUp(self):
        # Stuff Happens

    def tearDown(self):
        # more Stuff

    @mock.patch('storages.backends.s3boto.S3BotoStorage', FileSystemStorage)
    def test_file_stuff(self):
        self.assertMagicPonies(True)

Some gotchas - make sure you have a sane MEDIA_ROOT setup in the settings. as of django 1.4, you can't use the testing context manager to override MEDIA_ROOT, so you need a separate settings config for it (https://code.djangoproject.com/ticket/17787) This was fixed in 1.6. Also, make sure your upload_to works in normal filesystem, or you will get permission errors.

this doesn't work for me, storage in the FileField is still set to the original definition, not FileSystemStorage — Peyman, May 18 '17 at 15:39

score 0 · Answer 4 · answered Aug 27 '12 at 22:29

0

I would propose to use the standard Django Storage for testing, where you can a define custom path for storage and cleanup that path in your test suite once your done. Both the storage and the path can be set in the settings and overwritten for testing.

answered Aug 27 '12 at 22:29

schacki

9,401
5
29
32

2

I should have mentioned above, I'm using a custom Storage class and setting it just for this model in the field: models.FileField(storage=CustomS3Storage(...)) -- so swapping out the settings won't really work. – erikcw Aug 28 '12 at 15:41
@erikcw : you can still overwrite your model's FileField storage in the TestCase setUp (or at the tests module level etc). – bruno desthuilliers Sep 04 '12 at 17:27
@brunodesthuilliers how? – Nick Brady Jul 26 '21 at 20:52
1

@NickBrady the file storage is an attribute of the model's FileField, you just have to reassign another storage instance to the model's FileField, ie `MyModel.file.storage = SomeOtherStorage()`. – bruno desthuilliers Jul 28 '21 at 07:35

score 0 · Answer 5 · answered Jul 13 '23 at 05:43

I know this is a rather old thread, but Django 4.2 now offers a much simpler way with the InMemoryStorage. You just have to override the settings to declare it, and all media file accesses will be mocked in memory:

@override_settings(STORAGES={
    "default": {
        "BACKEND": "django.core.files.storage.memory.InMemoryStorage",
    },
})
class StorageTest(TestCase):
    """Any upload test will only use memory here"""
    ...

Mocking Django Storages Model ImageField backend S3

5 Answers5