4

I'm writing a Scala library to make querying an paginated JSON API easier. Each API call returns an object that looks something like this:

{
  "count": 100,
  "current_page": 1,
  "total_pages": 2,
  "records": [
    ...
  ]
}

I'd like to have a function that returned some sort of iterator like MyIterator[Record]. Are there any standard ways to do this in the Scala world, perhaps even constructs in the standard library that could help me?

I'm using lift-json generally for my JSON parsing, if that's helpful.

Thanks.

pr1001
  • 21,727
  • 17
  • 79
  • 125

1 Answers1

6

If you have an Iterator[Iterator[A]] you can use flatten to produce an Iterator[A] which chains all the nested Iterators together:

scala> Iterator(Iterator(1, 2), Iterator(3, 4)).flatten.toList
res0: List[Int] = List(1, 2, 3, 4)

Combined with one of the factory methods on the Iterator companion object, you could probably turn this into a one-liner:

Iterator.range(1, pageCount).map(i: Int => fetchPage(i)).flatten

It might need to be a bit more complex if you can only get the page count by retrieving a page... something like:

val (firstPage, pageCount) = getFirstPageWithCount()
firstPage ++ (Iterator.range(2, pageCount).map(fetchPage).flatten)
David Winslow
  • 8,472
  • 1
  • 31
  • 27
  • Thanks, David, that's very helpful. The API I'm targeting does only give the page count with the first page, so I'll have to use the second approach. Is there any way to only fetch subsequent pages if necessary? That is, a sort of lazy initialization? – pr1001 Jan 29 '11 at 23:06
  • Actually, the example I gave you is lazy already. (Try `Iterator.range(1, 10).map { x => println(x); x } .take(3).toList` to verify this yourself.) – David Winslow Jan 29 '11 at 23:14
  • Touché! Very cool, thanks! By the way, when might a Stream be more appropriate? – pr1001 Jan 29 '11 at 23:31
  • http://stackoverflow.com/questions/1527962/difference-between-iterator-and-stream-in-scala discusses the differences quite nicely, I think. – David Winslow Jan 29 '11 at 23:43