Calling arbitrary number of WS.url().get() in sequence

Question

I have a List[String] of URLs that I want to load and process (parse, store to database) in sequence.

I found only fixed-length examples, like:

    def readUrls = Action {
    implicit request => {
        implicit val context = scala.concurrent.ExecutionContext.Implicits.global

        val url1 = "http://some-website.com"
        val url2 = "http://other-website.com"

        Async {
            for {
                result1 <- WS.url(url1).get()
                result2 <- WS.url(url2).get()
            } yield {
                Ok(result1.body + result2.body)
            }
        }
}

But instead of url1 and url2, I need to process this puppy:

val urls = List("http://some-website.com", "http://other-website.com")

Thanks a bunch for any tips and advice!

Answer 1

If you want to chain Futures together arbitrarily in sequence, foldLeft ought to do the job:

urls.foldLeft(Future.successful[String]("")){ case (left, nextUrl) =>
    left.flatMap{ aggregatedResult =>
        WS.url(nextUrl).get().map( newResult =>
            aggregatedResult + newResult.body
        )
    }
}

Since you're just combining the request bodies together, I gave the foldLeft an initial value of a Future empty String, which each step in the fold will then add on the next response body.

Answer 2

def executeUrls(urls: List[String]): Future[String] = {
  urls.foldLeft(Future(""))((accumulator, url) => {
      accumulator.flatMap(acc => {
        WS.url(url).get().map(response => {
          acc + response.body
        })
      }
  })
}

This should be what you're looking for, note that it returns a new Future.

Edit: apparently LimbSoup was faster.