0

I am using flask and simplejson and have configured a custom json encoder to convert dicts to json.

Custon Encoder:

gen = []
class CustomJSONEncoder(json.JSONEncoder):

  def default(self, o):
    if isinstance(o, datetime):
      return o.strftime('%Y-%m-%d %H:%M:%S')
    if isinstance(o, date):
      return o.strftime('%Y-%m-%d')
    if isinstance(o, np.int64):
      return int(o)
    if isinstance(o, decimal.Decimal):
      a = (str(o) for o in [o])
      gen.append(a)
      return a

  def _iterencode(self, o, markers=None):
    if isinstance(o, np.int64):
      return int(o)
    if isinstance(o, decimal.Decimal):
      return (str(o) for o in [o])    
    return super(CustomJSONEncoder, self)._iterencode(o, markers)


def json_encode(data):
  return CustomJSONEncoder().encode(data)

Suppose I have a variable as b = [{"os": Decimal('7'), "num": 77777}].

If I run json_encode(b). This results in '[{"os": null, "num": 77777}]'. The Decimal value got converted to a null.

Now I have fixed this by changing the way the custom encoder handles Decimals by just returning a float(o). While this works, I want to understand whats wrong with the original version of the encoder.

I have tried capturing the generator object in a global variable gen and iterated over it. I can see the value 7 coming correctly.

Full ipython console log:

In [93]: bb
Out[93]: [{'os': Decimal('7'), 'num': 77777}]

In [94]: gen = []
    ...: class CustomJSONEncoder(json.JSONEncoder):
    ...:
    ...:   def default(self, o):
    ...:     if isinstance(o, datetime):
    ...:       return o.strftime('%Y-%m-%d %H:%M:%S')
    ...:     if isinstance(o, date):
    ...:       return o.strftime('%Y-%m-%d')
    ...:     if isinstance(o, np.int64):
    ...:       return int(o)
    ...:     if isinstance(o, decimal.Decimal):
    ...:       a = (str(o) for o in [o])
    ...:       gen.append(a)
    ...:       return a
    ...:
    ...:
    ...:   def _iterencode(self, o, markers=None):
    ...:     if isinstance(o, np.int64):
    ...:       return int(o)
    ...:     if isinstance(o, decimal.Decimal):
    ...:       return (str(o) for o in [o])
    ...:     return super(CustomJSONEncoder, self)._iterencode(o, markers)
    ...:
    ...:
    ...: def json_encode(data):
    ...:   return CustomJSONEncoder().encode(data)
    ...:

In [95]: json_encode(bb)
Out[95]: '[{"os": null, "num": 77777}]'

In [96]: gen[0]
Out[96]: <generator object CustomJSONEncoder.default.<locals>.<genexpr> at 0x7f45cdb37990>

In [97]: next(gen[0])
Out[97]: '7'

Where exactly is the issue ? Why is null being returned for Decimal ?

leoOrion
  • 1,833
  • 2
  • 26
  • 52
  • Because `(str(o) for o in [o])` is a generator. That isn't handled by JSON, so it too is passed to your `default` method, at which point, the function terminates without returning anything and returns `None`, which is treated as `null`. – juanpa.arrivillaga Jun 07 '21 at 06:01
  • so, *why* are you returning a generator? What did you *expect* to happen?? – juanpa.arrivillaga Jun 07 '21 at 06:01
  • note, if you had returned `json.JSONEncoder.default(self, o)` as is the documented best practice, then it wouldn't have failed silently. – juanpa.arrivillaga Jun 07 '21 at 06:03
  • https://stackoverflow.com/a/1960649/4983469. I was using this as a reference. – leoOrion Jun 07 '21 at 06:24
  • It was working well for around a year now. We just upgraded one of our machines and ran into this issue. Strange thing is... Some of our old boxes still work with the original version and is returning back proper values. Im trying to figure out what changed and why it does not work anymore. – leoOrion Jun 07 '21 at 06:24
  • AFAIKT if this ever worked it worked on "accident", i.e. based on non-public parts of the API and implementation details. This is why you shouldn't rely on implementation details, and rather, rely on the documented interfaces for these sorts of things – juanpa.arrivillaga Jun 07 '21 at 06:27

0 Answers0