isinstance not working for Decimal in AppEngine

Question

I have an instance of decimal.Decimal which comes from an SQLAlchemy query. As I need to serialize the object, I have created a JSON serializer to deal with the Decimal:

import decimal
class AlchemyEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, decimal.Decimal):
            return str(obj)
        return json.JSONEncoder.default(self, obj)

The unfortunate thing is that the isinstance(obj, decimal.Decimal) does not return a True for the instance, even though (using pdb in the default method above):

obj.__class__ # => <class 'decimal.Decimal'>
blah = decimal.Decimal()
blah.__class__ # => <class 'decimal.Decimal'>
isinstance(obj, decimal.Decimal) # => False
isinstance(blah, decimal.Decimal) # => True
isinstance(obj, obj.__class__) # => True

I did check that the module both instances refer to is the same module:

import inspect
inspect.getfile(obj.__class__) # => '/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/decimal.pyc'
inspect.getfile(blah.__class__) # => '/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/decimal.pyc'

I would really love to understand why this is not working!

EDIT

It turns out that the issue only occurs when running under AppEngine's dev_appserver.py environment. A simple:

isinstance(db.session.execute('SELECT amount FROM model LIMIT 1').fetchone()[0], decimal.Decimal)

returns False when making a request via the AppEngine dev_appserver and True when run from the console.

You seem to have an indentation error in your code ... the `return json.JSONEncoder.default(...)` should probably be in the `default` function. — mgilson, Dec 22 '16 at 00:11
@mgilson Cheers, it was - just went wrong when copying it onto SO. — Aert, Dec 22 '16 at 00:15
Any possibility for us to be able to reproduce this strange behavior? — jmd_dk, Dec 22 '16 at 00:33
Yeah, we might need a bit more info. I can't reproduce this with an sqlite db, but sqlite doesn't handle decimal natively, and sqlalchemy has to do the conversion from floating point. Are you using MySQL, PostgreSQL, ...? — Andrew Guy, Dec 22 '16 at 01:29
I tried to generate a reproducible piece of code, and in doing so I found the issue only occurs when running under the AppEngine's `dev_appserver.py`. Not sure how it's doing it, but it seems to be messing with the classes somehow. — Aert, Dec 22 '16 at 01:35
You might be able to work around it by making some sort of dummy query to get a reference to the Decimal class it has somehow imported. — BrenBarn, Dec 22 '16 at 02:00
@BrenBarn Interesting idea, I'll look into that, thanks! For now I added a `if str(obj.__class__) == ""` which I'm not proud of. Still don't get how it would report the same module / file for both instances though, if it's importing a different one somehow... — Aert, Dec 22 '16 at 02:11
@Aert: It is possible if the module is findable in multiple ways through `sys.path`. See for instance [this question](http://stackoverflow.com/questions/4798589/what-could-cause-a-python-module-to-be-imported-twice). It could be that GAE is doing some troublesome path manipulations. You could try peeking into `sys.modules` to see if you can see the module in there twice. — BrenBarn, Dec 22 '16 at 02:18
Cheers, I had a peek, in both cases it's there only once (and both cases the same module path) — Aert, Dec 22 '16 at 02:31
Very odd. I've never used GAE so I don't think I can offer any more specific suggestions. You would have to hunt through to see where else it's doing the import and how. — BrenBarn, Dec 22 '16 at 02:59
@BrenBarn It is very strange indeed. Thanks, really appreciate your and others' suggestions. — Aert, Dec 22 '16 at 03:02
Why not use `import numbers if isinstance(obj, numbers.Number)`? That checks for other types (int, float, long, complex) as well. — GAEfan, Dec 22 '16 at 05:22

score 5 · Accepted Answer · edited May 23 '17 at 12:16

Ran into this today, and came across an old mailing list post that discussed this.

Turns out this question has also been addressed on stackoverflow previously as well. Pasting the answer here for ease of access / reduced stackoverflow server load:

It appears that the decimal.Decimal class is patched somewhere in the Google App Engine SDK (or the module is reloaded), and this is done between the MySQL conversion library importing decimal and you importing the same.

Luckily, we can work around this by updating MySQL conversion table:

from MySQLdb.constants import FIELD_TYPE
from MySQLdb.converters import conversions
import decimal

conversions[FIELD_TYPE.DECIMAL] = conversions[FIELD_TYPE.NEWDECIMAL] = decimal.Decimal

That's it; the above code resets the MySQL class and your JSON encoder type check succeeds.

The alternative would be for you to look for the class MySQLdb is using:

class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, MySQLdb.converters.conversions[MySQLdb.constants.FIELD_TYPE.DECIMAL]):
            return float(o)
        return super(DecimalEncoder, self).default(o)

isinstance not working for Decimal in AppEngine

1 Answers1