Celery Error: numpy array is not JSON serializable

Question

I'm using flask server with celery and redis. The error occurred when .apply_async() is called. The numpy array is part of visualizing the output of a keral neural network model. I do know that there is a way to convert keras model to json. My main problem lies in the fact that I do not know when or how celery performs the conversion, and I do not have control over it.

Here is my code:

@celery.task(bind=True)
def celery_createDirectoryAndSaveNNOutput(self, pInput, ID, filename, layersToShow, model):
    layer_outputs = [layer.output for layer in model.layers[1:]]
    viz_model = Model(input=model.input, output=layer_outputs)
    features = viz_model.predict(pInput)

    layerOutputs = {}
    folderName = "static/"+ID+"_"+filename

    if not os.path.exists(folderName):
        os.makedirs(folderName)

    for layerIndex in layersToShow:
        images = getFeatureMapImages(features[int(layerIndex)])
        layerOutputs[layerIndex] = []
        for i in range(0, len(images)):
            path = folderName+"/layer"+str(int(layerIndex))+"_"+str(i)+".jpg"
            cv2.imwrite(path, images[i])
            layerOutputs[layerIndex].append(path)
        self.update_state(state='PROGRESS', meta={'current': 0, 'total': 10,"status":filename})

    return {'current': i, 'total': len(layersToShow),'status': "temp"}


@app.route("/nnvisualisation_uploadMultipleImages", methods=["POST"])
def nnvisualisation_uploadMultipleImages():
    uploaded_files = request.files.getlist("file[]")
    weight = request.form.get("weight")
    ID = request.form.get("ID")

    layersToShow = [5]
    modelName = "VGG16"

    preds = {}
    path = os.path.join(STATIC_PATH, uploaded_files[0].filename)
    uploaded_files[0].save(os.path.join(STATIC_PATH, uploaded_files[0].filename))
    pInput, result = preTrainedModel[modelName](path)
    #ERROR HERE:
    task = celery_createDirectoryAndSaveNNOutput.s( pInput=pInput, ID=ID, filename=uploaded_files[0].filename, layersToShow=layersToShow, model=getModel(modelName)).apply_async(serializer='json')
    ...


    return jsonify({}), 202, {'Location': url_for('taskstatus',task_id=task.id)}

I've tried all available serializer yaml:

EncodeError: cannot represent an object: keras.engine.training.Model object at 0x10fdf26d0>

pickle:

EncodeError: Can't pickle type 'module': attribute lookup builtin.module failed

msgpack:

EncodeError: can't serialize array([[[[-103.93900299, -107.77899933, -123.68000031],... , dtype=float32) (numpy array)

json:

EncodeError: array([[[[-103.93900299, -107.77899933, -123.68000031],... , dtype=float32) (numpy array) is not JSON serializable

Any comment or suggestion is greatly appreciated. Thank you.

`json` is a string format compatible with `javascript`. It encodes dictionaries, lists and strings. Other `python` classes have to have to 'serialize' themselves into one of those structures. `numpy` arrays don't automatically do that,though there are tools that can help. Do some searching about `keras` and `json`. — hpaulj, Feb 10 '17 at 17:30
Thank you for your comment. I do know that there is a way to convert keras model to json. My main problem lies in the fact that I do not know when or how celery performs the conversion, and I do not have control over it. — matchifang, Feb 10 '17 at 20:57

score 2 · Answer 1 · edited May 23 '17 at 12:17

2

My main problem lies in the fact that I do not know when or how celery performs the conversion, and I do not have control over it.

There dose exist a way to control conversion. You can register customized json serializer which can dump numpy array.

Refer this doc Serializers

And there is a good example.

edited May 23 '17 at 12:17

Community

1
1

answered Feb 19 '17 at 12:50

gzc

8,180
8
42
62

score -1 · Answer 2 · answered Feb 10 '17 at 17:50

-1

https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model

# save as JSON
json_string = model.to_json()

though that saves just the architecture not the weights etc.

In any case you need to explore the methods provided by keras.

answered Feb 10 '17 at 17:50

hpaulj

221,503
14
230
353

Thank you for your comment. The problem is that the conversion is done by celery, so I do not have control over how the model is changed to json. – matchifang Feb 10 '17 at 20:26
Could you please advise me where I should put that line of code at? I don't know where celery serializes it. Thank you. – matchifang Feb 13 '17 at 21:41

Celery Error: numpy array is not JSON serializable

2 Answers2