SQLAlchemy mapping table with non-ascii columns to class

Question

item = Table('Item', metadata, autoload=True, autoload_with=engine, encoding = 'cp1257')

class Item(object):
    pass

from sqlalchemy.orm import mapper
mapper(Item, item)

I get error:

line 43, in <module>
    mapper(Item, item) 
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\__init__.py", line 890, in mapper
    return Mapper(class_, local_table, *args, **params)
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\mapper.py", line 211, in __init__
    self._configure_properties()
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\mapper.py", line 578, in _configure_properties
    setparent=True)
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\mapper.py", line 618, in _configure_property
    self._log("_configure_property(%s, %s)", key, prop.__class__.__name__)
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\mapper.py", line 877, in _log
    (self.non_primary and "|non-primary" or "") + ") " + 
  File "C:\Python27\lib\site-packages\sqlalchemy\util.py", line 1510, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "C:\Python27\lib\site-packages\sqlalchemy\sql\expression.py", line 3544, in description
    return self.name.encode('ascii', 'backslashreplace')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xeb in position 7: ordinal not in range(128)

I am connecting to MSSQL. table autoload seems to work. I only get this error while trying to map. Thank you all for help!

I currently use 'cp1257' encoding, reading column names works with this seting, with utf8 I can not even autoload my table. — Atoc, Apr 29 '11 at 20:38

score 3 · Accepted Answer · answered Apr 29 '11 at 12:30

Mapping the table to a class creates mapped properties on the class. The properties have the same name of the columns, by default. Since python 2.x only allows ascii identifiers, that fails if you have non-ascii column names.

The only solution I can think of is to give the identifiers a different name when mapping the table to a class.

The example below does that. Note that I'm creating the table on the code for simplicity, so anyone can run the code without having existing table. But you could do the same with a reflected table.

#-*- coding:utf-8 -*-

import sqlalchemy as sa
import sqlalchemy.orm

engine = sa.create_engine('sqlite://', echo=True) # new memory-only database
metadata = sa.MetaData(bind=engine)

# create a table. This could be reflected from the database instead:
tb = sa.Table('foo', metadata, 
    sa.Column(u'id', sa.Integer, primary_key=True),
    sa.Column(u'nomé', sa.Unicode(100)),
    sa.Column(u'ãéìöû', sa.Unicode(100))
)    
tb.create()

class Foo(object):
    pass

# maps the table to the class, defining different property names 
# for some columns:
sa.orm.mapper(Foo, tb, properties={
    'nome': tb.c[u'nomé'],
    'aeiou': tb.c[u'ãéìöû']
})

After that you can use Foo.nome to refer to the nomé column and Foo.aeiou to refer to the ãéìöû column.

This works, however I was thinking about automagic, because this solution at least for me defeats using autoload. And if someone adds new column my application breaks. So maybe I am better off using "exclude_properties" and "include_properties" or not using autoload — Atoc, Apr 29 '11 at 19:48
[link](http://stackoverflow.com/questions/1207457/convert-unicode-to-string-in-python-containing-extra-symbol) this link helps anyone who wants to automate the above solution. — Atoc, Apr 29 '11 at 20:58
The same can be achieved with declarative style as shown in the docs: http://docs.sqlalchemy.org/en/latest/orm/mapper_config.html#automating-column-naming-schemes-from-reflected-tables — Akasha, Jul 24 '14 at 21:51
`autoload` has more problems, it tries to load all relations and for big database it takes lots of time (>10min) — Atoc, Jan 03 '17 at 23:20

score 1 · Answer 2 · answered Nov 17 '11 at 02:56

I faced the same problem and finally managed to do it replacing table['column'].key after autoloading it, just make all your table classes inherit this one and then modify the column name replacement in mapTo method or override manually the desired names with a dictionary and columns_descriptor method. I don't know if this is not the right way to do it but after searching for hours is the best aproach I've got.

class SageProxy(object):

@classmethod
def ismapped(cls, table_name=None):
    if mappings:
        if table_name:
            if mappings.has_key(table_name):
                tmap=mappings[table_name]
                if tmap.has_key('class'):
                    tclass=tmap['class']
                    if tclass is cls:
                        return True
        else:
            for m in mappings:
                if cls is m['class']:
                    return True
    return False

@classmethod
def mappingprops(cls):
    #override this to pass properties to sqlalchemy mapper function
    return None

@classmethod
def columns_descriptors(cls):
    #override this to map columns to different class properties names
    #return dictionary where key is the column name and value is the desired property name
    return {}

@classmethod
def mapTo(cls, table_name, map_opts=None):
    if not cls.ismapped(table_name):
        tab_obj=Table(table_name,sage_md,autoload=True)
        for c in tab_obj.c:
            #clean field names
            tab_obj.c[c.name].key=c.key.replace(u'%',u'Porcentaje').replace(u'ñ',u'ny').replace(u'Ñ',u'NY').replace(u'-',u'_') 
        for k,v in cls.columns_descriptors():
            if tab_obj.c[k]:
                tab_obj.c[k].key=v
        mapper(cls, tab_obj, properties=cls.mappingprops())
        mappings[table_name]={'table':tab_obj,'class':cls}
    return cls

I expect it will be usefull

score 1 · Answer 3 · answered Feb 24 '12 at 16:21

I found that I could do this with a simple addition to my reflected class:

metadata = MetaData(bind=engine, reflect=True)
sm = sessionmaker(bind=engine)

class tblOrders(Base):
    __table__ = metadata.tables['tblOrders']
    meter = __table__.c['Meter#']

meter is now mapped to the underlying Meter# column, which allows this code to work:

currOrder = tblOrders()
currOrder.meter = '5'

Without the mapping, python sees it as a broken statement becase Meter followed by a comment does not exist in the object.

SQLAlchemy mapping table with non-ascii columns to class

3 Answers3

Linked