0

Looking for a way to decrease the use of memory in python, I created this code

from pympler.asizeof import asizeof


class Reduce(type):
    def __new__(cls, name, bases, attrs):
        attrs['__dict__'] = {}
        return super().__new__(cls, name, bases, attrs)


class WithReduce(metaclass=Reduce):
    def __init__(self):
        self.a = 1

class Normal:
    def __init__(self):
        self.a = 1

print(asizeof(WithReduce())) # 122
print(asizeof(Normal())) # 240

I was able to reduce the memory usage of the class by almost 50% excluding the content of __dict__ in the metaclass.

In some tests I realized that, when adding new attributes in the WithReduce, they do not take up space in memory and are not stored in __dict__. However, in the Normal they occupy memory and are stored in __dict__

a = WithReduce()
b = Normal()

print(asizeof(a)) # 122
print(asizeof(b)) # 240

a.foo = 100
b.foo = 100

print()
print(asizeof(a)) # 112
print(asizeof(b)) # 328

print()
print(a.__dict__) # {}
print(b.__dict__) # {'a': 1, 'foo': 100}

Knowing this, I had two doubts.

  • Why didn't WithReduce take up more memory by adding a variable to it?

  • Where is the variable is added in WithReduce?

ellandor
  • 79
  • 1
  • 1
  • 9
  • 1
    I'm not sure you saved any memory; you just somehow shadowed the actual instance `__dict__` in a way that prevents `pympler` from seeing it. Adding an entry to `attrs` just created a new class attribute; it wouldn't affect the *instance* attribute present in an instance of a class. – chepner Dec 17 '20 at 20:47
  • Read up on `__dict__` a little. https://stackoverflow.com/questions/19907442/explain-dict-attribute You can't really just delete `__dict__` and expect normal behaviour. It's pretty fundamental to having a functioning objects. – Macattack Dec 17 '20 at 20:53
  • That makes sense. Is there any way to make python not to use `__dict__`, just like `__slots__` does? – ellandor Dec 17 '20 at 20:54
  • Maybe back up a bit and ask why you have so many Python objects that carrying around their `__dict__` is a memory concern. Usually when you end up in this situation it's better to use array(s) of data rather than many individual Python instances each carrying their own namespace. – wim Dec 17 '20 at 21:11
  • This responds to my comment: https://stackoverflow.com/a/9654228/14243840 – ellandor Dec 17 '20 at 21:15
  • Unrelated to your specific question, but you should take a look at `__slots__`. – Kapocsi Dec 17 '20 at 21:16

1 Answers1

0

In a normal class, __dict__ is a descriptor that provides access to an instance's attributes:

>>> Normal.__dict__['__dict__']
<attribute '__dict__' of 'Normal' objects>
>>> Normal.__dict__['__dict__'].__get__(b)
{'a': 1, 'foo': 100}

However, you replaced (or at least shadowed) that with an ordinary dictionary in WithReduce:

>>> WithReduce.__dict__['__dict__']
{}

I'm not really sure how attribute assignment works if you modify a class like this. setattr apparently will not use your explicit __dict__ attribute, but rather is hard-coded(?) to use the descriptor, for which there may be a reference around somewhere.

chepner
  • 497,756
  • 71
  • 530
  • 681
  • 1
    Attribute access doesn't go through the `__dict__` descriptor unless the attribute is `__dict__`; it finds an object's instance dict using the `tp_dictoffset` field in the class struct. (You can trace the code through `PyObject_GenericSetAttr` down to `_PyObject_GetDictPtr` in [`Objects/object.c`](https://github.com/python/cpython/blob/v3.9.1/Objects/object.c#L1358); `PyObject_GenericSetAttr` is the default `__setattr__` implementation.) – user2357112 Dec 17 '20 at 21:12
  • Nice; I didn't bother digging into the C code, because I usually just get lost and confused when I try. – chepner Dec 17 '20 at 21:13