How to serve, reference, and save static files in Flask/python app.py or views.py files, avoiding FileNotFound errors

Question

Similar to this question, but there's some follow ups I have. How to serve static files in Flask

The main issue I'm having in adapting the solution from that previous post is that I need to use matplotlib's .savefig() function, which just takes in a string for a path to save to. So I need to know what string to pass it.

I remember having a lot of issues with static files when I was first working on my Flask application locally. I don't know how I'm supposed to correctly reference the path of static files from my app.py/views.py python file. I understand using url_for() in html and what not, but referencing the static files in python hasn't been as straightforward.

My flask app has the following structure. The root directory is called Neuroethics_Behavioral_Task

I have a file called experiment.py and that file is basically a views.py or app.py file. But I can't seem to find the correct way to reference static files.

Here's the experiment.py code.

import functools

from flask import (
    Blueprint, flash, g, redirect, render_template, request, session, url_for, Flask
)
from werkzeug.security import check_password_hash, generate_password_hash


app = Flask(__name__, instance_relative_config=True, static_url_path='') # creates the flask instance

bp= Blueprint('experiment', __name__)


@app.route('/')
@app.route('/index')
def index():
    return render_template("welcome.html")

@app.route('/experiment_app', methods=['GET', 'POST'])
def exp_js_psych_app():
    total_trials = int(request.form['trials'])
    import random 
    import string 
    user_id = ''.join([random.choice(string.ascii_letters 
                + string.digits) for n in range(8)]) 
    user_id = str(user_id)
    stim_metadata = generate_stim_for_trials(total_trials, user_id)
    return render_template("index.html", data={"user_id": str(user_id), "total_trials":total_trials, "lst_of_stim_jsons":stim_metadata})
def generate_stim_for_trials(trials, uid):
    import matplotlib
    import matplotlib.pyplot as plt
    import pandas as pd
    import numpy as np
    import json
    import os
    json_dict = {}
    json_dict["stimuli path info"] = []
    lst_of_jsons_for_stimuli = []
    # Unused but problematic static reference.
    stim_metadata_path = "/static/textdata/stimulus_data" + str(uid) + ".json"
    # Another problematic static reference.
    stimuli_path = '/static/images/stimulus_img' + 1 + ".png"
    df = pd.DataFrame({"Risk":[25], "No Effect" : [50], "Gain":[25]}) 
    df.plot(kind='barh', legend = False, stacked=True, color=['#ed713a', '#808080', '#4169e1'])
    plt.axis('off')
    plt.gca().set_axis_off()
    plt.subplots_adjust(top = 1, bottom = 0, right = 1, left = 0, hspace = 0, wspace = 0)
    plt.ylim(None,None)
    plt.autoscale()
    plt.margins(0,0)

    plt.savefig(stimuli_path) # this fails

Something to note is that passing in the static_url_path parameter to the Flask() call isn't really working as intended, I thought that this would establish the root as the place to first look for my static files but the fact that this code is still failing suggests otherwise, since my /static/ directory is clearly in the root folder.

Otherwise as I said... I need to use plt.savefig() which takes in a path, and I still can't figure out how I'm supposed to write this path.

I have managed to get static files to be working prior to this point. I had to change the two references above to static files to the following:

stimuli_path = 'Neuroethics_Behavioral_Task/static/images/stimulus_img' + "1" + ".png"
stim_metadata_path = "Neuroethics_Behavioral_Task/static/textdata/stimulus_data" + str(uid) + ".json"

In other words, I could get static files before by including the root directory in the path.

and this at least got me by... Until I tried to deploy my app onto Heroku. Then I started to get errors that Neuroethics_Behavioral_Task/static/images/ wasn't a recognized path... so now I feel like I need to actually start referencing static files correctly, and I'm not sure how to at this point.

I've tried variations (ex: images/stimulusimg1.png) and tried changing the static_url_path parameter (ex: static_url_path='/static/), but these have all just caused my Flask application to crash locally saying FileNotFound.

Another important follow up question I have...

This application is going to be on an online experiment posted on Amazon Mechanical Turk. Images are supposed to be preloaded by a framework I'm using called JsPsych. But posts like this and the one I first included encourage you to serve files using NGINX What exactly does 'serving static files' mean?

I have no idea how Nginx or wsgi work. Should I still bother to try to get all of my static files served this way, and if so how do I start?

So in summary the questions I'm asking are

• I need to save static files in my experiment.py/views.py/apps.py file, and so I need to figure out the string that actually denotes the path to my static images. How do I do that? Do I actually need to modify the static_url_path variable?
• Why have I previously been successful with referencing the root directory in my paths to static files
• My images are supposed to be preloaded by a javascript framework I'm using, JsPsych. Should I still go through the trouble of setting up nginx and wsgi configuration?

edit: url_for() isn't working either.

stimuli_in_static = "images/stimulus_img" + "1" + ".png"
stimuli_path = url_for('static', filename=stimuli_in_static)

This still results in a FileNotFound error at the .savefig() call,

FileNotFoundError: [Errno 2] No such file or directory: '/static/images/stimulus_img1.png'

Answer 1

As per my comments, because Heroku doesn't support persistent filesystem storage, the recommended method is to use S3. To do that you'll need to have plot.fig write to a Bytes IO buffer, then upload that to S3. This process is covered in this SO thread.

Either way, here's an answer on the static file stuff regardless, which is permissable on Heroku provided the static files in question are actually part of the repo (javascript and CSS files, etc)...

---

static_url_path is designed to change the URL which static files are available at. That affects the frontend URL path from which static files are available.

With the following config static files are available at http://example.com/apple/ regardless of where they are on the filesystem:

app = Flask(__name__, static_url_path='apple')

If you want to define where Flask looks on the filesystem for static files, use static_folder instead. In your case the application is at app_dir/experiment.py and your static directory is at the top level. So you need to provide that argument relative to where experiment.py is:

app = Flask(__name__, static_folder='../static')

Personally I try to do away with this whole headache and create a directory structure which conforms to what Flask expects by default:

project
├── experiment.py
├── static
│   └── some.js
└── templates
    └── some.html

---

One other tip regarding your code:

def generate_stim_for_trials(trials, uid):
    import matplotlib
    import matplotlib.pyplot as plt
    import pandas as pd
    import numpy as np
    import json
    import os

This is inefficient, because all of these imports will be loaded each time that function is called.

You really want to put those imports at the top of the file, so they load when the app server runs, and only then.:

import matplotlib
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import json
import os

# ...

def generate_stim_for_trials(trials, uid):
    json_dict = {}
    # ...

Answer 2

Consider flask's documentation on how to organize flask applications. The first file structure they talk about is the 'simple module' pattern, the second file structure is the more complicated 'package' pattern. https://flask.palletsprojects.com/en/1.1.x/tutorial/layout/

I was originally trying to impelement a "simple module" pattern. The contents of app_dir were empty. Instead, Neuroethics_Behavioral_Task was my project directory, and 1 level inside of it were my experiment.py, __init__.py, /static/ and /templates/ folders.

I was able to get past these FileNotFound errors by switching to the package-based organization pattern for my flask app. So the file structure now looks like

In this scheme, Neuroethics_Behavioral_Task is still the project directory. But now there's app_dir which contains __init.py__, experiment.py, /static/ and /template/ all in the same directory.

When I made this change, I was able to rewrite most of the save() statements in experiment.py to something like this.

# inside of generate_stim_for_trials()
csv_path = "app_dir/static/textdata/" + subj_id + "_data.csv"

In other words, all I had to do was use the app_dir prefix.

This worked both on my local machine and it worked once I pushed the code to heroku. I'm still using this "package" based organization pattern since it's what flask at least recommends for complex applications.

On another hand, V25 made a very crucial point that Heroku does not provide permanent storage, which may totally change your plans if you're deploying your stuff to heroku like I am. I'm glad I mentioned that I intended to do that.