2

I'm using the code from this thread to limit my download rate.

How do I incorporate partial downloads resuming with the rate limiting code? The examples I've found use urlopen instead of urlretrieve, and the RateLimit class depends on urlretrieve.

I'd like to have an external function that controls the partial downloading, without having to change the RateLimit class:

from throttle import TokenBucket, RateLimit

def retrieve_limit_rate(url, filename, rate_limit):
    """Fetch the contents of urls"""
    bucket = TokenBucket(10*rate_limit, rate_limit)

    print "rate limit = %.1f kB/s" % (rate_limit,)

    print 'Downloading %s...' % filename
    rate_limiter = RateLimit(bucket, filename)
    #
    # What do I put here to allow resuming files?
    #
    return urllib.urlretrieve(url, filename, rate_limiter)
Community
  • 1
  • 1
Lionel
  • 3,188
  • 5
  • 27
  • 40
  • possible duplicate of [How to resume download in PYTHON, using urlretrieve function??](http://stackoverflow.com/questions/3581296/how-to-resume-download-in-python-using-urlretrieve-function) and [Download file using urllib in Python with the wget -c feature](http://stackoverflow.com/questions/2021519) – moinudin Dec 26 '10 at 09:35
  • Not duplicates because: urllib.urlretrieve can limit the rate using class RateLimit. urllib2.urlopen can resume files, but can't limit rate. Need a way to combine both a rate limit and a file resume – Lionel Dec 26 '10 at 10:11

1 Answers1

4

May be able to use PyCurl instead:

def curl_progress(total, existing, upload_t, upload_d):
    try:
        frac = float(existing)/float(total)
    except:
        frac = 0
    print "Downloaded %d/%d (%0.2f%%)" % (existing, total, frac)

def curl_limit_rate(url, filename, rate_limit):
    """Rate limit in bytes"""
    import pycurl
    c = pycurl.Curl()
    c.setopt(c.URL, url)
    c.setopt(c.MAX_RECV_SPEED_LARGE, rate_limit)
    if os.path.exists(filename):
        file_id = open(filename, "ab")
        c.setopt(c.RESUME_FROM, os.path.getsize(filename))
    else:
        file_id = open(filename, "wb")

    c.setopt(c.WRITEDATA, file_id)
    c.setopt(c.NOPROGRESS, 0)
    c.setopt(c.PROGRESSFUNCTION, curl_progress)
    c.perform()
Lionel
  • 3,188
  • 5
  • 27
  • 40