Django: calling .only() on my model causing infinite loop?

Question

I am using .only to fetch the required fields from my model, it seems like my __init__ method causing a inifinite loop when calling only on this model, and this is my model:

class PodcastEpisode(models.Model):
    audio_metadata = models.JSONField(null=True)
    featured_artists = models.ManyToManyField(to=User, related_name='featured_artists')
    podcast_series = models.ForeignKey(to=PodcastSeries, on_delete=models.CASCADE, null=False)
    published = models.BooleanField(default=False)
    published_at = models.DateTimeField(blank=True, null=True)

    _original_audios = None # To store current data
    _published = None

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        # self._original_audios = self.audio_metadata
        # self._published = self.published

When I commented these lines self._original_audios = self.audio_metadata and self._published = self.published, it doesn't cause inifinite loop. I am not sure how this is happening even if I have included audio_metadata in my .only() fields. This is my query

PodcastEpisode.objects\
            .filter(id__in=id_list).prefetch_related(*prefetches).only(*['id' 'audio_metadata'])

Please suggest me how do I use .only() and where should I place these _original_audios and _published variables.

For reference this is the whole stacktrace:

 File "/Users/dev/Desktop/dev/Podsack_backend/mediacontent/models.py", line 152, in __init__
    self._original_audios = self.audio_metadata
                            ^^^^^^^^^^^^^^^^^^^
  File "/Users/dev/Library/Caches/pypoetry/virtualenvs/narratave-9vDc9ea1-py3.11/lib/python3.11/site-packages/django/db/models/query_utils.py", line 182, in __get__
    instance.refresh_from_db(fields=[field_name])
  File "/Users/dev/Library/Caches/pypoetry/virtualenvs/narratave-9vDc9ea1-py3.11/lib/python3.11/site-packages/django/db/models/base.py", line 707, in refresh_from_db
    ).filter(pk=self.pk)
      ^^^^^^^^^^^^^^^^^^
  File "/Users/dev/Library/Caches/pypoetry/virtualenvs/narratave-9vDc9ea1-py3.11/lib/python3.11/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dev/Library/Caches/pypoetry/virtualenvs/narratave-9vDc9ea1-py3.11/lib/python3.11/site-packages/django/db/models/query.py", line 1420, in filter
    return self._filter_or_exclude(False, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dev/Library/Caches/pypoetry/virtualenvs/narratave-9vDc9ea1-py3.11/lib/python3.11/site-packages/django/db/models/query.py", line 1438, in _filter_or_exclude
    clone._filter_or_exclude_inplace(negate, args, kwargs)
  File "/Users/dev/Library/Caches/pypoetry/virtualenvs/narratave-9vDc9ea1-py3.11/lib/python3.11/site-packages/django/db/models/query.py", line 1445, in _filter_or_exclude_inplace
    self._query.add_q(Q(*args, **kwargs))
  File "/Users/dev/Library/Caches/pypoetry/virtualenvs/narratave-9vDc9ea1-py3.11/lib/python3.11/site-packages/django/db/models/sql/query.py", line 1532, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dev/Library/Caches/pypoetry/virtualenvs/narratave-9vDc9ea1-py3.11/lib/python3.11/site-packages/django/db/models/sql/query.py", line 1562, in _add_q
    child_clause, needed_inner = self.build_filter(
                                 ^^^^^^^^^^^^^^^^^^
  File "/Users/dev/Library/Caches/pypoetry/virtualenvs/narratave-9vDc9ea1-py3.11/lib/python3.11/site-packages/django/db/models/sql/query.py", line 1478, in build_filter
    condition = self.build_lookup(lookups, col, value)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dev/Library/Caches/pypoetry/virtualenvs/narratave-9vDc9ea1-py3.11/lib/python3.11/site-packages/django/db/models/sql/query.py", line 1289, in build_lookup
    lookup_class = lhs.get_lookup(lookup_name)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dev/Library/Caches/pypoetry/virtualenvs/narratave-9vDc9ea1-py3.11/lib/python3.11/site-packages/django/db/models/expressions.py", line 377, in get_lookup
    return self.output_field.get_lookup(lookup)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dev/Library/Caches/pypoetry/virtualenvs/narratave-9vDc9ea1-py3.11/lib/python3.11/site-packages/django/db/models/query_utils.py", line 216, in get_lookup
    found = self._get_lookup(lookup_name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dev/Library/Caches/pypoetry/virtualenvs/narratave-9vDc9ea1-py3.11/lib/python3.11/site-packages/django/db/models/query_utils.py", line 203, in _get_lookup
    return cls.get_lookups().get(lookup_name, None)
           ^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded in comparison

I added the stacktrace @ybl, and I also saw it's running queries endlessly — suvodipMondal, Aug 28 '23 at 05:50
So I found the answer here https://stackoverflow.com/a/65557252/13060649 — suvodipMondal, Aug 29 '23 at 03:18
I feel a cleaner way would be to override `refresh_from_db`. Check my updated answer if it helps. — ybl, Aug 29 '23 at 03:33

ybl · Answer 1 · 2023-08-29T03:33:04.097

The Django ORM (Object-Relational Mapping) makes use of lazy loading. When you try to access self.audio_metadata and self.published inside __init__, Django hasn't actually finished initializing the object, and tries to retrieve these values from the database. This retrieval triggers the __init__ function again, leading to an infinite loop.

Instead of trying to save these values at initialization, you could override Django's save method like so:

    def save(self, *args, **kwargs):
        # if self.pk is not None, then the instance is currently in the database
        if self.pk is not None:
            # retrieve the instance from the database
            old_values = PodcastEpisode.objects.get(pk=self.pk)
            self._original_audios = old_values.audio_metadata
            self._published = old_values.published
        super().save(*args, **kwargs)

Update

To avoid the extra DB call above, you can override the refresh_from_db function in addition to save

    def refresh_from_db(self, using=None, fields=None, **kwargs):
        super().refresh_from_db(using, fields, **kwargs)
        self._original_audios = self.audio_metadata
        self._published = self.published

    def save(self, *args, **kwargs):
        if self._published is None:
            self._original_audios = self.audio_metadata
            self._published = self.published
        super().save(*args, **kwargs)

But with other methods it was not happening, like it's only happening with `only()` and `defer()` — suvodipMondal, Aug 28 '23 at 07:29
I don't know the other methods where this behaviour is not observed, but it looks like `only()` / `defer()` triggeres `__init__` at some point. Also, you are correct that will trigger a query, sorry but I don't have a good enough solutions for that at the moment. — ybl, Aug 28 '23 at 07:55

Django: calling .only() on my model causing infinite loop?

1 Answers1