Running an async function fetching a site in a loop

Question

Hey guys i am scraping a website getting an image url for a movie/ I have these function currently getting the image which is working just fine outside a loop

 async function getPosterDB(imdbID) {
  const posterdbURL = "https://www.movieposterdb.com/search?q=";
  let poster = "";
  try {
    return fetch(`${posterdbURL}${imdbID}`)
      .then((response) => response.text())
      .then((body) => {
        const $ = cheerio.load(body);
        poster = $(
          "body > div.section > div > div > div.col-md-9 > div.pt-3 > div.row.mgrid > div > a > img"
        ).attr("src");
        if (poster) {
          poster = poster.replace("posters", "xl");
          poster = poster.replace("/s", "/xl");
        }
      });
  } catch (error) {
    console.log(error);
  }
  return poster;
}

The thing is when i try to get the url running through a loop . i get undefined or a promise Now i searched on the web and found out i should use either a setTimeout function to delay the function or either Promise.all() , maybe i am missing something out The code which is the function is here.

async function advancedSearchMovies(searchTerm) {
  const movies = [];
  let images = [];
  try {
    return await fetch(`${MovieAdvancedUrl}${searchTerm}`)
      .then((response) => response.text())
      .then(async (body) => {
        const $ = cheerio.load(body);
        $("#main > div > div.lister.list.detail.sub-list > div >").each(
          async function (i, elem) {
            let imdbID = $(elem)
              .find("div.lister-item-image.float-left > a")
              .attr("href")
              .match(/title\/(.*)\//)[1];
          
            )
              let poster = getPosterDB(imdbID)
              const movie = {
                imdbID: imdbID,
                title: title,
                poster
              };

              movies.push(movie);
            }
          }
        );
        const result = Promise.all(movies).then(data => { return data});
        return result;
      });
  } catch (error) {
    console.log(error);
  }
}

But i still get a promise back , any idea ?

{ imdbID: 'tt1185834', title: 'Star Wars: The Clone Wars', yearRun: '2008', ratingIMDB: '5.9', ratingMeta: '35', summary: "After the Republic's victory on Christophsis, Anakin and his new apprentice Ahsoka Tano must rescue the kidnapped son of Jabba the Hutt. Political intrigue complicates their mission.", poster: Promise { pending } }

PROBLEM FIXED : I just made the function return a promise

async function getPosterDB(imdbID) {
return new Promise((resolve, reject) => {
  fetch(`${posterdbURL}${imdbID}`)
    .then((response) => response.text())
    .then((body) => {
      const $ = cheerio.load(body);
      poster = $(
        "body > div.section > div > div > div.col-md-9 > div.pt-3 > div.row.mgrid > div > a > img"
      ).attr("src");
      if (poster) {
        poster = poster.replace("posters", "xl");
        poster = poster.replace("/s", "/xl");
      }
      setTimeout(() => {
        resolve(poster);
      }, 1000);
    });
});
}

and then when i call it like this:

const images = await Promise.all(
          movies.map((element) =>
            getPosterDB(element.imdbID).then((data) => {
              return data;
            })
          )
        );
        movies.forEach((element, index) => {
          element.poster = images[index];
        });
        return movies;
      });

The second version looks fine, and as you observed it returns a Promise. So just call `.then(...)` on the result. — Robin Zigmond, Jun 11 '20 at 20:32
PS the `.then(data => { return data})` does nothing and therefore can be removed. — Robin Zigmond, Jun 11 '20 at 20:33
adding another .then ( (final) => console.log(final)) gives undefined — Stelios V., Jun 11 '20 at 20:47

Bergi · Answer 1 · 2020-06-12T17:33:17.417

You need to put promises inside the array that you're passing to Promise.all. You're only pushing movie objects, and you're doing that asynchronously, so when you call Promise.all the array is still unpopulated.

Also I'd recommend to avoid using .then() calls when working with async/await:

async function advancedSearchMovies(searchTerm) {
  try {
    const response = await fetch(`${MovieAdvancedUrl}${searchTerm}`);
    const body = await response.text();
    const $ = cheerio.load(body);
    const promises = Array.from($("#main > div > div.lister.list.detail.sub-list > div >")).map(async elem => {
      const imdbID = $(elem)
        .find("div.lister-item-image.float-left > a")
        .attr("href")
        .match(/title\/(.*)\//)[1];
      const poster = await getPosterDB(imdbID)
      const movie = {
        imdbID,
        title,
        poster,
      };
      return movie;
    });
    const movies = await Promise.all(promises);
    return movies;
  } catch (error) {
    console.log(error);
  }
}

Notice that you should not use try/catch syntax if you cannot really handle the error. Logging it and returning undefined is probably not what the caller of your function expects. Better let the exception bubble.

Regarding getPosterDB, avoid the Promise constructor antipattern!

async function getPosterDB(imdbID) {
  const response = await fetch(`${posterdbURL}${imdbID}`);
  const body = await response.text();
  const $ = cheerio.load(body);
  let poster = $(
    "body > div.section > div > div > div.col-md-9 > div.pt-3 > div.row.mgrid > div > a > img"
  ).attr("src");
  if (poster) {
    poster = poster.replace("posters", "xl");
    poster = poster.replace("/s", "/xl");
  }
  return poster;
}

Still getting either a promise back and when i resolve it i get undefined.Tried also to pass an array of all the promises made during the loop still nothing. — Stelios V., Jun 12 '20 at 16:25

Running an async function fetching a site in a loop

1 Answers1