SqlAlchemy not using utf8 charset even if specified

Question

I'm using Flask with Flask-SqlAlchemy with MySQL and my connection string does have the charset-utf8 parameter on my connection string.

I'm also using reflection and my Model is specified as this:

class Value(db.Model):
    __bind_key__ = 'values'
    __tablename__ = 'values'

I've also tried the trick of putting # -*- coding:utf8 -*- inside my files but I'm still getting these stacktraces:

Traceback (most recent call last):
  File "/home/numkem/src/sd/venv/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/numkem/src/sd/venv/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/home/numkem/src/sd/venv/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/numkem/src/sd/venv/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/numkem/src/sd/venv/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/numkem/src/sd/venv/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/numkem/src/sd/venv/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/numkem/src/sd/venv/lib/python2.7/site-packages/flask_debugtoolbar/__init__.py", line 125, in dispatch_request
    return view_func(**req.view_args)
  File "/usr/lib/python2.7/cProfile.py", line 149, in runcall
    return func(*args, **kw)
  File "/home/numkem/src/sd/application/views/player.py", line 54, in player_show
    return render_template('player/show.html', **locals())
  File "/home/numkem/src/sd/venv/lib/python2.7/site-packages/flask/templating.py", line 128, in render_template
    context, ctx.app)
  File "/home/numkem/src/sd/venv/lib/python2.7/site-packages/flask/templating.py", line 110, in _render
    rv = template.render(context)
  File "/home/numkem/src/sd/venv/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/home/numkem/src/sd/venv/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/numkem/src/sd/application/templates/player/show.html", line 1, in top-level template code
    {% extends "base.html" %}
  File "/home/numkem/src/sd/application/templates/base.html", line 62, in top-level template code
    {% block body %}{% endblock %}
  File "/home/numkem/src/sd/application/templates/player/show.html", line 27, in block "body"
    <td>{{ field.values }}</td>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 16: ordinal not in range(128)

Why is the charset ignored on connection? Shouldn't it be set by the connection string? Maybe it's just not something that possible/supported when using reflections.

Thanks!

score 4 · Accepted Answer · edited May 23 '17 at 10:31

Turns out the problem was not really coming from SqlAlchemy but more from Jinja2 itself since it wasn't seeing that I wanted to use utf8 encoding througout the application. the documentation states that the default encoding is set to ascii if nothing is specified.

This answer shows the solution to my problem.

The fix is to add this to your first code:

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

I'm not 100% familiar with python but I'm assuming this might have to do with using a WSGI application instead of a standard script so the # -*- coding:utf8 -*- trick doesn't work.

score 0 · Answer 2 · answered Mar 17 '14 at 17:08

0

This error means that you pass non-unicode strings in your templates as context, because Jinja2 supports only unicode.

You should read this material about encoding.

answered Mar 17 '14 at 17:08

Ellochka Cannibal

1,750
2
19
31

The data comes from SqlAlchemy, I'm able to see it appear correctly outside of the template. Starting to think it might have to do with Jinja itself rather than SqlAlchemy – numkem Mar 17 '14 at 17:31

SqlAlchemy not using utf8 charset even if specified

2 Answers2