Python - Keras API server to process image on trained custom model

Question

I've followed this tutoria to create a custom object YoloV3 Keras model: https://momoky.space/pythonlessons/YOLOv3-object-detection-tutorial/tree/master/YOLOv3-custom-training

Model works perfectly fine, my next goal is to create a Python Flask API witch is capable to process Image after upload it. I've started modify the Code here for image detection

That's my added code:

@app.route('/api/test', methods=['POST'])
def main():
    img = request.files["image"].read()
    img = Image.open(io.BytesIO(img))
    npimg=np.array(img)
    image=npimg.copy()
    image=cv2.cvtColor(image,cv2.COLOR_BGR2RGB)
    #cv2.imshow("Image", image)
    #cv2.waitKey()
    cv2.imwrite('c:\\yolo\\temp.jpg', image)
    image = 'c:\\yolo\\temp.jpg'
    yolo = YOLO()
    r_image, ObjectsList = yolo.detect_img(image)
    #response = {ObjectsList}
    response_pikled = jsonpickle.encode(ObjectsList)
    #yolo.close_session()
    return Response(response=response_pikled, status=200, mimetype="application/json")
app.run(host="localhost", port=5000)

So my problem is that it works only on first iteration, when I upload a new image I receive following error:

File "C:\Users\xxx\Anaconda3\envs\yolo\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
    run_metadata_ptr)
  File "C:\Users\xxx\Anaconda3\envs\yolo\lib\site-packages\tensorflow\python\client\session.py", line 1095, in _run
    'Cannot interpret feed_dict key as Tensor: ' + e.args[0])
TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("Placeholder:0", shape=(3, 3, 3, 32), dtype=float32) is not an element of this graph.

This is the original static part of the code:

if __name__=="__main__":
    yolo = YOLO()
    image = 'test.png'
    r_image, ObjectsList = yolo.detect_img(image)
    print(ObjectsList)
    #cv2.imshow(image, r_image)
    cv2.imwrite('detect.png', r_image)

    yolo.close_session()

Things that really confuse me is how to load the model when the application start, and execute detection every time a new image is posted. Thank you

UPDATE

in the construtor part there's a referenced Keras backend session:

 **def __init__(self, **kwargs):
        self.__dict__.update(self._defaults) # set up default values
        self.__dict__.update(kwargs) # and update with user overrides
        self.class_names = self._get_class()
        self.anchors = self._get_anchors()
        self.sess = K.get_session()
        self.boxes, self.scores, self.classes = self.generate()**

After addinga K.clear_session it works for multiple series request:

 @app.route('/api/test', methods=['POST'])
    def main():
        img = request.files["image"].read()
        img = Image.open(io.BytesIO(img))
        npimg=np.array(img)
        image=npimg.copy()
        image=cv2.cvtColor(image,cv2.COLOR_BGR2RGB)
        #cv2.imshow("Image", image)
        #cv2.waitKey()
        cv2.imwrite('c:\\yolo\\temp.jpg', image)
        image = 'c:\\yolo\\temp.jpg'
        yolo = YOLO()
        r_image, ObjectsList = yolo.detect_img(image)
        #response = {ObjectsList}
        response_pikled = jsonpickle.encode(ObjectsList)
        #yolo.close_session()
        K.clear_session()
        return Response(response=response_pikled, status=200, mimetype="application/json")

Will be possible to avoid model, anchors and classes needs to be loaded at every computation avoiding this:

ogs/000/trained_weights_final.h5 model, anchors, and classes loaded.
127.0.0.1 - - [27/Dec/2019 22:58:49] "?[37mPOST /api/test HTTP/1.1?[0m" 200 -
logs/000/trained_weights_final.h5 model, anchors, and classes loaded.
127.0.0.1 - - [27/Dec/2019 22:59:08] "?[37mPOST /api/test HTTP/1.1?[0m" 200 -
logs/000/trained_weights_final.h5 model, anchors, and classes loaded.
127.0.0.1 - - [27/Dec/2019 22:59:33] "?[37mPOST /api/test HTTP/1.1?[0m" 200 

-

Answer 1

Inside YOLO constructor try adding this:

from keras import backend as K
K.clear_session()

Answer 2

I've managed to get this up and running as a prototype. I've uploaded a repo: vulcan25/image_processor which implements this all.

The first thing I investigated was the functionality of the method YOLO.detect_img in that code from the tutorial. This method takes a filename, which is immediately handled by cv2.imread in the original code: #L152-L153. The returned data from this is then processed internally by self.detect_image (note the difference in spelling) and the result displayed with cv2.show.

This behaviour isn't good for a webapp and I wanted to keep everything in memory, so figured the best way to change that functionality was to subclass YOLO and overide the detect_img method, making it behave differently. So in processor/my_yolo.py I do something like:

from image_detect import YOLO as stock_yolo

class custom_yolo(stock_yolo):
    def detect_img(self, input_stream):

        image = cv2.imdecode(numpy.fromstring(input_stream, numpy.uint8), 1)

        original_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        original_image_color = cv2.cvtColor(original_image, cv2.COLOR_BGR2RGB)

        r_image, ObjectsList = self.detect_image(original_image_color)
        is_success, output_stream  = cv2.imencode(".jpg", r_image)

        return is_success, output_stream

Note: In a later decision, I pulled image_detect.py into my repo to append K.clear_session(). It would have been possible to put the above mod in that file also, but I've stuck with subclassing for that part.

This accepts a stream, then uses cv2.imencode (source) and cv2.imdecode (source) in place of imshow and imread respectively.

We can now define a single a single function which will in turn run all the image processing stuff. This separates that part of the code (and dependencies) from your flask app which is good:

yolo = custom_yolo() # create an object from the custom class defined above.

def process(intput_stream):

     start_time = time.time()

     is_success, output_stream = yolo.detect_img(input_stream)
     io_buf = io.BytesIO(output_stream)

     print("--- %s seconds ---" % (time.time() - start_time))

     return is_success, io_buf.read()

From Flask we can call this in the same way, where we already have the stream of the uploaded file available to us as: request.files['file'].read() (which is actually a method of the werkzeug.datastructures.FileStorage object, as I've documented elsewhere).

As a side-note, this function could be run from the terminal. If you're launching my repo you'd need to do this in the processor container (see the docker exec syntax at the end of my README)...

from my_yolo import process
with f.open('image.jpg', 'rb') as f:
    is_sucess, processed_data = process(f.read())

Then the result written to a file:

with f.open('processed_image.jpg', 'wb' as f):    
    f.write(processed_data)

Note that my repo actually has two separate flask apps (based on another upload script I put together which implements dropzone.js on the frontend).

I can run in two modes:

1. processor/src/app.py: Accessible on port 5001, this runs process directly (incoming requests block until the processed data is returned).
2. flask/src/app.py: Accessible on port 5000, this creates an rq job for each incoming request, the processor container then runs as a worker to process these requests from the queue.

Each app has its own index.html which does its own unique implementation on the frontend. Mode (1) writes images straight to the page, mode (2) adds a link to the page, which when clicked leads to a separate endpoint that serves the image (when processed).

The major difference is how process is invoked. With mode (1) processor/src/app.py:

from my_yolo import process

if file and allowed_file(file.filename):
    # process the upload immediately
    input_data = file.read()
    complete, data = process(input_data)

As mentioned in a comment, I was seeing pretty fast conversions with this mode: ~1.6s per image on CPU. This script also uses a redis set to maintain a list of uploaded files, which can be used for validation on the view endpoint further down.

In mode (2) flask/src/app.py:

from qu import image_enqueue

if file and allowed_file(file.filename):
            input_data = file.read()

            job = img_enqueue(input_data)
            return jsonify({'url': url_for('view', job_id=job.id)})

I've implemented a separate file flask/src/qu.py which implements this img_enqueue function, which ultimately loads the process function from flask/src/my_yolo.py where it is defined as:

def process(data): pass

This is an important destinction. Normally with rq the contents of this function would be defined in the same codebase as the flask service. In fact, I've actually put the business logic in processor/src/my_yolo.py which allows us to detach the container with the image processing dependencies, and ultiately host this somewhere else, as long is it shares a redis connection with the flask service.

Please have a look at the code in the repo for further info, and feel free to log an issue against that repo with any further queries (or if you get stuck). Be aware I may introduce breaking changes, so you may wish to fork.

I've tried to keep this pretty simple. In theory this could be edited slightly to support a different processor/Dockerfile which handles any processing workload, but the same frontend allowing you to submit any type of data from a stream: images, CSV, other text, etc.

Things that really confuse me is how to load the model when the application start, and execute detection every time a new image is posted. Thank you

You'll notice when you run this mode (1) it is perfect, in that the dependencies load when the flask server boots (~17.s) and individual image processing takes ~1s. This is ideal, although probably leads to higher overall memory usage on the server, as each WSGI worker requires all the dependencies loaded.

When run in mode (2) - where processing is passed to rq workers, the libraries are loaded each time an image is processed, so it's much slower. I will try to fix that, I just need to investigate how to pre-load the libraries in the rq deployment; I was close to this before but that was around the time I stumbled with the K.clear_session() problem, so haven't had time to retest a fix for this (yet).