2

I'm using requests to query a paginated API.

This answer already seems pretty un-pythonic to me but my case is even worse because the API doesn't have a last page parameter, it only has a last URL parameter, so I have to do

url = 'http://my_url'
result = request(url)
data = result['datum']
while not result['last_url'] == url:
  url = result['next_page']
  result = request(url)
  data += result['datum']

This doesn't seem pythonic, is there a more elegant way?

Community
  • 1
  • 1
user291699
  • 45
  • 4
  • 2 spaces indentation? – Peter Wood Jan 15 '16 at 12:37
  • Maybe you should post your question in http://codereview.stackexchange.com/ – Salva Jan 15 '16 at 12:49
  • It looks ok to me. I assume `result['datum']` is a string. If so, it's a little more efficient to accumulate the data in a list and then use `.join` to concatenate the strings in one hit. – PM 2Ring Jan 15 '16 at 12:53
  • 1
    @Salva: Fair call, however this question in its current state is **not** suitable for CodeReview. Please see [A guide to Code Review for Stack Overflow users](http://meta.codereview.stackexchange.com/questions/5777/a-guide-to-code-review-for-stack-overflow-users) for details. – PM 2Ring Jan 15 '16 at 12:56
  • Answered my own question, I think this is way nicer: http://pastebin.com/nvgQrbaq – user291699 Jan 15 '16 at 13:18
  • @user291699 How many pages can your solution return? It's probably [at most 997](http://stackoverflow.com/questions/3323001/maximum-recursion-depth). – Peter Wood Jan 15 '16 at 13:34
  • You're right, I was looking up whether Python had tail recursion as you posted that. But I still prefer http://pastebin.com/JrHquXYR to the original solution. – user291699 Jan 15 '16 at 13:37
  • No, Python can't optimize tail call recursion, and as Peter Wood mentions, Python has a recursion limit (which can be adjusted if you really need to exceed the default limit). So it's generally best to avoid recursion in Python unless you're doing something that is naturally suited to recursion, eg processing recursive data structures, like trees. – PM 2Ring Jan 16 '16 at 05:31

1 Answers1

2

I would write a generator function which yields pages:

def pages(url):
    page = {'last_url': None,
            'next_page': url}
    while page['last_url'] != url:
        url = page['next_page']
        page = request(url)
        yield page

Then you can use it as so:

data = ''.join([page['datum'] for page in pages(url)])
Peter Wood
  • 23,859
  • 5
  • 60
  • 99