3

I am trying to understand how to handle a http.client.IncompleteRead Error in the code below. I handle the error using the idea in this post. Basically, I thought it might just be the server limiting the number of times I can access the data, but strangely I get a HTTP Status Code of 200 sometimes, but still the code below ends up returning a None type. Is this because zipdata = e.partial is not returning anything when the Error comes up?

def update(self, ABBRV):
    if self.ABBRV == 'cd':
        try:

            my_url = 'http://www.bankofcanada.ca/stats/results/csv'
            data = urllib.parse.urlencode({"lookupPage": "lookup_yield_curve.php",
                                 "startRange": "1986-01-01",
                                 "searchRange": "all"})

            binary_data = data.encode('utf-8')
            req = urllib.request.Request(my_url, binary_data)
            result = urllib.request.urlopen(req)
            print('status:: {},{}'.format(result.status, my_url))
            zipdata = result.read()
            zipfile = ZipFile(BytesIO(zipdata))

            df = pd.read_csv(zipfile.open(zipfile.namelist()[0]))
            df = pd.melt(df, id_vars=['Date'])

            return df

        #In case of http.client.IncompleteRead Error
        except http.client.IncompleteRead as e:
            zipdata = e.partial

Thank You

Community
  • 1
  • 1
user131983
  • 3,787
  • 4
  • 27
  • 42
  • Can you add yourimport statements? (e.g. `from zipfile import ZipFile`) It's a major pet peeve of mine when I have to spend 5 minutes trying to work out the libraries a questioner is using just so that I can run the code and help them. – maxymoo Jun 26 '15 at 03:59

1 Answers1

0

Hmm... I've run your code 20 times without getting an incomplete read error, why don't you just retry in the case of an incomplete read? Or on the other hand if your IP is being blocked, then it would make sense that they're not returning you anything. Your code could look something like this:

maxretries = 3
attempt = 0
while attempt < maxretries:
   try:
      #http access code goes in here
   except http.client.IncompleteRead:
      attempt += 1
   else:
      break
Crutchcorn
  • 199
  • 5
  • 14
maxymoo
  • 35,286
  • 11
  • 92
  • 119
  • Its strange. Its only sometimes the Error pops up. Previously the code was fine. Also, did you mean add this to the code: `#In case of http.client.IncompleteRead Error except http.client.IncompleteRead as e: df = self.update(ABBRV)[1] return df` – user131983 Jun 26 '15 at 13:37
  • 1
    Added an `else` to make sure the loop terminated. I know this question is old, but it helped me and I wanted to contribute back – Crutchcorn Jan 17 '17 at 21:48