How to break an asynchronous loop based on a string?

Question

Problem:

I am trying to pull every record from each record type in NetSuite using a restlet asynchronously in Python. I have a working synchronous solution, but I need the extra speed. Each record in every record type has an index assigned to it, so I want to iterate on those indexes in order to successfully pull every record. However, I cannot seem to figure out a dynamic solution using these indexes to iterate and successfully resolve my loop.

async def make_one_request(url: str, num: int):
    headers = {
        'Authorization': authHeaderSetup(num),
        'Content-Type': "application/json",
        'cache-control': "no-cache",
    }

    params = '?script=' + script + '&deploy=' + deploy
    params += '&record_type=' + recordType + '&index=' + str(num) + '&date=' + date

    # No more than 4 concurrent workers
    async with limit:
        print(f"Making request {num}")
        r = await client.get(url + params, headers=headers)

        if limit.locked():
            print("Loading...")
            await asyncio.sleep(0.00001)

        if r.json() == 'end':
            asyncio.get_event_loop().stop()

        if r.status_code == HTTPStatus.OK:
            return r

    raise ValueError(
        f"Unexpected Status: Http status code is {r.status_code}.",
    )


async def make_many_requests(url: str, count: int) -> list[httpx.Response]:
    tasks = []
    for num in range(count):
        task = asyncio.create_task(make_one_request(url, num))
        tasks.append(task)

    results = await asyncio.gather(*tasks)

    print("\n")
    print("Final result:")
    print("==============\n")
    for result in results:
        print(result.json())
    asyncio.get_event_loop().stop()


if __name__ == "__main__":
    start = time.time()
    run(make_many_requests(url, count=61))
    end = time.time()
    print(f'It took {end - start} seconds')

My current solution is to manually create the range based on my prior knowledge of the json data and its size. But obviously, this is in NO WAY a dynamic solution. On the SuiteScript side, I have conveniently placed the keyword "end" just at the end of each array of json to signal that this is the end of that iteration. My question is... how can I "break" the loop at the "end" keyword and gather my tasks to move onto the next iteration.

Does this answer your question? [Asyncio.gather vs asyncio.wait](https://stackoverflow.com/questions/42231161/asyncio-gather-vs-asyncio-wait) — ti7, Oct 07 '22 at 03:59

score 0 · Answer 1 · answered Oct 07 '22 at 03:58

If I understand you correctly, you could use the built-in asyncio.wait() function to do this:

async def make_many_requests(url: str, count: int) -> list[httpx.Response]:
    tasks = []
    for num in range(count):
        task = asyncio.create_task(make_one_request(url, num))
        tasks.append(task)

    # Wait for all tasks to be done
    done, pending = await asyncio.wait(tasks)

    # If there are any pending tasks, cancel them
    if pending:
        for task in pending:
            task.cancel()

    results = [task.result() for task in done]

    print("\n")
    print("Final result:")
    print("==============\n")
    for result in results:
        print(result.json())
    asyncio.get_event_loop().stop()

My question is how will it know the tasks are done? I want the loop to run continuously until the request outputs the keyword "end", then break and gather the task and move onto the next record type. Sorry if I am not making any sense... first time coding with asyncio — Ryan Lebato, Oct 07 '22 at 04:08
The await asyncio.wait(tasks) statement will wait for all tasks to be done before moving on. — JDong, Oct 07 '22 at 04:08

How to break an asynchronous loop based on a string?

1 Answers1