Python UnicodeDecodeError: 'utf8' codec can't decode byte... unexpected code byte

Question

Python newbie's journey to build his first webapp (app link: http://contractpy.appspot.com/ - it's just an experimental app).

Following an advice of a stackoverflow user, I start to use a template system: Jinja2 (I'm using Python 2.6), but now I'm stucked with this error:

2012-06-17 11:44:39 Running command: "['C:\\Python26\\pythonw.exe', 'C:\\Program Files (x86)\\Google\\google_appengine\\dev_appserver.py', '--admin_console_server=', '--port=8084', 'C:\\Users\\CG\\Documents\\udacity\\contract']"
Warning: You are using a Python runtime (2.6) that is older than the production runtime environment (2.7). Your application may be dependent on Python behaviors that have changed and may not work correctly when deployed to production.
WARNING  2012-06-17 14:44:41,993 datastore_file_stub.py:518] Could not read datastore data from c:\users\cg\appdata\local\temp\dev_appserver.datastore
WARNING  2012-06-17 14:44:42,005 dev_appserver.py:3401] Could not initialize images API; you are likely missing the Python "PIL" module. ImportError: No module named _imaging
INFO     2012-06-17 14:44:42,025 dev_appserver_multiprocess.py:647] Running application dev~contractpy on port 8084: http://localhost:8084
INFO     2012-06-17 14:44:42,025 dev_appserver_multiprocess.py:649] Admin console is available at: http://localhost:8084/_ah/admin
WARNING  2012-06-17 14:44:47,171 py_zipimport.py:139] Can't open zipfile C:\Python26\lib\site-packages\setuptools-0.6c11-py2.6.egg-info: IOError: [Errno 13] file not accessible: 'C:\\Python26\\lib\\site-packages\\setuptools-0.6c11-py2.6.egg-info'
ERROR    2012-06-17 14:44:47,674 wsgi.py:189] 
Traceback (most recent call last):
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 187, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 225, in _LoadHandler
    handler = __import__(path[0])
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_import_hook.py", line 676, in Decorate
    return func(self, *args, **kwargs)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_import_hook.py", line 1858, in load_module
    return self.FindAndLoadModule(submodule, fullname, search_path)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_import_hook.py", line 676, in Decorate
    return func(self, *args, **kwargs)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_import_hook.py", line 1722, in FindAndLoadModule
    description)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_import_hook.py", line 676, in Decorate
    return func(self, *args, **kwargs)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_import_hook.py", line 1665, in LoadModuleRestricted
    description)
  File "C:\Users\CG\Documents\udacity\contract\main.py", line 60, in <module>
    class MainHandler(webapp2.RequestHandler):
  File "C:\Users\CG\Documents\udacity\contract\main.py", line 71, in MainHandler
    template = jinja_environment.get_template('index.html')
  File "C:\Program Files (x86)\Google\google_appengine\lib\jinja2\jinja2\environment.py", line 719, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "C:\Program Files (x86)\Google\google_appengine\lib\jinja2\jinja2\environment.py", line 693, in _load_template
    template = self.loader.load(self, name, globals)
  File "C:\Program Files (x86)\Google\google_appengine\lib\jinja2\jinja2\loaders.py", line 115, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "C:\Program Files (x86)\Google\google_appengine\lib\jinja2\jinja2\loaders.py", line 169, in get_source
    contents = f.read().decode(self.encoding)
  File "C:\Python26\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 3403: unexpected code byte
INFO     2012-06-17 14:44:47,713 dev_appserver.py:2891] "GET / HTTP/1.1" 500 -
INFO     2012-06-17 14:44:47,801 dev_appserver.py:2891] "GET /favicon.ico HTTP/1.1" 200 -
ERROR    2012-06-17 14:52:18,345 wsgi.py:189] 
Traceback (most recent call last):
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 187, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 225, in _LoadHandler
    handler = __import__(path[0])
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_import_hook.py", line 676, in Decorate
    return func(self, *args, **kwargs)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_import_hook.py", line 1858, in load_module
    return self.FindAndLoadModule(submodule, fullname, search_path)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_import_hook.py", line 676, in Decorate
    return func(self, *args, **kwargs)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_import_hook.py", line 1722, in FindAndLoadModule
    description)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_import_hook.py", line 676, in Decorate
    return func(self, *args, **kwargs)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_import_hook.py", line 1665, in LoadModuleRestricted
    description)
  File "C:\Users\CG\Documents\udacity\contract\main.py", line 60, in <module>
    class MainHandler(webapp2.RequestHandler):
  File "C:\Users\CG\Documents\udacity\contract\main.py", line 71, in MainHandler
    template = jinja_environment.get_template('index.html')
  File "C:\Program Files (x86)\Google\google_appengine\lib\jinja2\jinja2\environment.py", line 719, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "C:\Program Files (x86)\Google\google_appengine\lib\jinja2\jinja2\environment.py", line 693, in _load_template
    template = self.loader.load(self, name, globals)
  File "C:\Program Files (x86)\Google\google_appengine\lib\jinja2\jinja2\loaders.py", line 115, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "C:\Program Files (x86)\Google\google_appengine\lib\jinja2\jinja2\loaders.py", line 169, in get_source
    contents = f.read().decode(self.encoding)
  File "C:\Python26\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 3403: unexpected code byte
INFO     2012-06-17 14:52:18,378 dev_appserver.py:2891] "GET / HTTP/1.1" 500 -
INFO     2012-06-17 14:52:18,470 dev_appserver.py:2891] "GET /favicon.ico HTTP/1.1" 200 -

I'm using PyScripter and it's set to encode in utf-8:

I did a search on stackoverflow and found many issues on it but I didn't understand what is going wrong in my case and how to fix it.

Thanks in advance for any help!

Here my code:

# -*- coding: utf-8 -*-
# -*- coding: utf-8 -*-
#!/usr/bin/env python
#
# Copyright 2007 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

import os

import webapp2

import jinja2

jinja_environment = jinja2.Environment(autoescape=True,
    loader=jinja2.FileSystemLoader(os.path.join(os.path.dirname(__file__), 'templates')))

import re

USER_RE = re.compile(r"^[a-zA-Z0-9_-]{3,20}$")
def valid_resident(resident):
    return USER_RE.match(resident)

PASS_RE = re.compile(r"^.{3,20}$")
def valid_SSN(SSN):
    return PASS_RE.match(SSN)

EMAIL_RE = re.compile(r"^[\S]+@[\S]+\.[\S]+$")
def valid_email(email):
    return EMAIL_RE.match(email)

def escape_html(s):
    for (i,o) in (("&","&"), (">",">"), ("<","<"), ('"','"')):
        s = s.replace(i,o)
        return s

import time

import datetime

def dateToday():
    today = datetime.datetime.today()
    todayDay = str(today.day)
    todayMonth = str(today.month)
    monthExt = {'1':' January ', '2':'February', '3':' March ', '4':'April', '5':'May', '6':'June', '7 ':' July ', '8':'August', '9':'September', '10':'October', '11':'November ', '12':'December'}
    todayYear = str(today.year)
    return(todayDay + ' of  ' + monthExtenso[todaymonth] + ' of ' + todayYear)

class MainHandler(webapp2.RequestHandler):
    def get(self):
       template_values = {"resident": "",
                                       "SSN": "",
                                       "driverLicense": "",
                                       "email":"",
                                       "resident_error": "",
                                       "SSN_error": "",
                                       "driverLicense_error": "",
                                       "email_error": ""}

    template = jinja_environment.get_template('index.html')
    self.response.out.write(template.render(template_values))

    def write_form(self, text_write):
       self.response.out.write(text_write)

    def post(self):
        resident_name = self.request.get(escape_html("resident"))
        user_nacionality = self.request.get("nacionality")
        user_SSN = self.request.get(escape_html('SSN'))
        user_email = self.request.get(escape_html('email'))
        user_driverLicense = self.request.get(escape_html('driverLicense'))
        resident_error = ""
        SSN_error = ""
        driverLicense_error = ""
        contract_type = self.request.get("contractType")
        owner = self.request.get("owner")
        witness  = self.request.get("witness ")

        if (resident_name and valid_resident(resident_name)) \
        and (user_SSN and valid_SSN(user_SSN)) \
        and ((not user_email) or (user_email and valid_email(user_email))):
            self.redirect('/yourcontract?resident=%s&nacionality=%s&SSN=%s&driverLicense=%s&email=%s&witness=%s&owner=%s' % (resident_name, user_nacionality, user_SSN, user_driverLicense, user_email,
witness, owner))
        else:
            if not valid_resident(resident_name):
                resident_error = "Oh no!!! this resident name isn't valid!"
            if not valid_SSN(user_SSN):
                SSN_error = "Oh no!!! SSN isn't valid!"
            if user_email and not valid_email(user_email):
                email_error = "Oh no!!! e-mail isn't valid!"
            self.write_form(form % {"resident":resident_name,
                                      "resident_error":resident_error,
                                      "SSN":user_SSN,
                                      "SSN_error":SSN_error,
                                      "driverLicense":user_driverLicense,
                                      "email":user_email,
                                      })

class yourcontractHandler(webapp2.RequestHandler):
    def get(self):
        geted_resident_name = self.request.get('resident')
        geted_user_nacionality = self.request.get("nacionality")
        geted_user_SSN = self.request.get('SSN')
        geted_user_email = self.request.get('email')
        geted_user_driverLicense = self.request.get('driverLicense')
        geted_resident_error = ""
        geted_SSN_error = ""
        geted_driverLicense_error = ""
        #geted_contract_type = self.request.get("contractType")
        geted_owner = self.request.get("owner")
        geted_witness  = self.request.get("witness")
        geted_contractType = self.request.get("contractType")



        self.response.out.write(yourcontract % {"resident":geted_resident_name,
                                      "nacionality":geted_user_nacionality,
                                      "SSN":geted_user_SSN,
                                      "SSN_error":geted_SSN_error,
                                      "driverLicense":geted_user_driverLicense,
                                      "email":geted_user_email,
                                      "witness ":geted_witness ,
                                      "owner":geted_owner,
                                      "contractType":geted_contractType
                                      })


app = webapp2.WSGIApplication([('/', MainHandler), ('/yourcontract', yourcontractHandler)],
                              debug=True)

Here my template code:

<!DOCTYPE html>

<html>
  <head>
    <title>Contract with Python</title>
    <style type="text/css">
      .label {text-align: right}
      .error {color: red}
    </style>

  </head>

  <body>
    <h2>Contract with Python</h2>
    <form method="post">
      <table>
        <tr>
          <td class="label">
            resident
          </td>
          <td>
            <input type="text" name="resident" 

value= "%(resident)s">
          </td>
          <td class="error"> 

%(resident_error)s

          </td>
        </tr>

         <tr>
          <td class="label">
            nacionality
          </td>
          <td>
            <input type="text" 

name="nacionality" value= " ">
          </td>
        </tr>

        <tr>
          <td class="label">
            License
          </td>
          <td>
            <input type="text" 

name="driverLicense" value="">
          </td>
        </tr>

        <tr>
          <td class="label">
            SSN
          </td>
          <td>
            <input type="text" name="SSN" 

value="%(SSN)s">
          </td>
          <td class="error"> %(SSN_error)s
          </td>
        </tr>


        <tr>
          <td class="label">
            Email (optional)
          </td>
          <td>
            <input type="text" name="email" 

value="%(email)s">
          </td>
          <td class="error"> %(email_error)s

          </td>
        </tr>
      </table>

<br>
        <td class="label">
          Contract Type
          <select name = "contractType">
              <option>House  Rental 

Contract</option>
              <option>Car Rental 

Contract</option>
              <option>Other</option>
          </select>
          </td>
<br>
<br>

<br>
          owner
          <select name = "owner">
              <option>House owner</option>
              <option>Car owner</option>
              <option>Other owner</option>
          </select>
<br>
<br>

<br>
          Witness
          <select name = "witness">
              <option>Carl Sagan</option>
              <option>Mahatma Gandhi</option>
          </select>
<br>
<br>

      <input type="submit">
    </form>
  </body>

</html>
"""

yourcontract = """
<html>
  <head>
    <title>Unit 2 Signup</title>
  </head>

  <body>

{{contractType}}

In consideration of the agreements of the 

Resident(s), known as: {{resident}} The owner 

hereby rents them the dwelling located at 

{{residentAdress}}, for the period commencing 

on the {{dateStarts}}, and monthly thereafter 

until the last day of {{dateEnds}}, at which 

time this Agreement is terminated. Resident

(s), in consideration of owners permitting 

them to occupy the above property, hereby 

agrees to the following terms:

RENT: To pay as rental the sum of $ 

{{rentalSum}} per month, due and payable in 

advance from the first day of every month. 

Failure to pay rent when due will result in 

the owner taking immediate legal action to 

evict the Resident from the premises and seize 

the security deposit.
LATE FEE: Rent received after the first of the 

month will be subject to a late fee of 10% 

plus (3.00) dollars per day.

ACCEPTED THIS {{dateContract}}, at 

{{localContract}}.

______________________________________________

_____
{{resident}} - Resident

______________________________________________

_____
{{owner}} – owner

______________________________________________

_____
{{witness}} – Witness

</body>
</html>

Answer 1

Your index.html file is not UTF-8 valid. Open the file with your code editor, and save as UTF-8. You can trace the error from the exception traceback:

  [...]
  File "C:\Users\CG\Documents\udacity\contract\main.py", line 71, in MainHandler
    template = jinja_environment.get_template('index.html')
  [...]
  File "C:\Program Files (x86)\Google\google_appengine\lib\jinja2\jinja2\loaders.py", line 169, in get_source
    contents = f.read().decode(self.encoding)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 3403: unexpected code byte

Answer 2

Your file encoding is declared as utf-8, but you have at least one part of it that doesn't make sense as utf-8 - this would tend to mean that it isn't in utf-8. The most likely solution to this is to ensure in whatever text editor or IDE you are using that files are saved as utf-8 - this will be different in each editor, but look for an 'Encoding' option somewhere.