Semaphore with async in Swift

Question

I want to do download say 10 files from internet, but only process max 3 at a time. It seems a Semaphore is a common solution. My internet call is an async function. I haven't found any code examples that combine these two.

Problem to solve:

async function to download 10 files
max 3 files processing at one time
wait until all files processed before returning from function

Problems I'm having:

semaphore.wait() does not work in async function
I need wait to wait until all 10 files have processed before leaving function

Goofy solution?

I'm calling Task.detached each loop to create new thread. Seems like a better solution should exist

Code sample:

func downloadAllFiles() async -> (String) {
    
    // Download 10 files, but max 3 at a time
    var urls = MArray<String>() ; for _ in 1 ... 10 { urls.add("https://google.com") }
    let semaphore = DispatchSemaphore(value: 3)
    
    for url in urls {
        semaphore.wait()    // wait if 3 already in progress ; doesn't work in async context
        Task.detached(priority: .high) {    // other priority options will general "lower QoS thread" warning
            let web = await MWeb.call(url: url)
            semaphore.signal()      // allow next download to start
        }
    }
    await Task.WhenAll(tasks) // *** C# code: need some way to wait until all tasks done before returning
    return ("")
}

If I run my code in non-async world, it seems to work, but I'd like to call such a function within in async process.

Can someone help with guidance?

Using a dispatch semaphore, or a dispatch anything, in connection with async/await is wrong and illegal. There is a way to do concurrent groups of three until all items are done, but what you are doing is not it. "Semaphore is a common solution" No it isn't. That would not be the right way if this were entirely dispatch queue code either. There is basically never a reason to use a semaphore. — matt, Dec 14 '22 at 23:59
Improper as it may be, if `downloadAllFiles` was hypothetically not async, this wouldn’t be illegal, right? The threads on the cooperative thread pool wouldn’t be blocked waiting for the semaphore, right? — Alexander, Dec 15 '22 at 01:25
@matt so what is the right thing then? Task groups don’t have a max concurrency setting, right? — Alexander, Dec 15 '22 at 01:25
@Alexander No but it's trivial to write a loop around the task group creation so that you hand a task group three things to do concurrently, wait until they are all done, then hand a task group the next three things to do concurrently, etc. — matt, Dec 15 '22 at 02:21
@Alexander: Check this: https://stackoverflow.com/a/70976324/1187415 — Martin R, Dec 15 '22 at 02:23
@matt ahhhhh that’s always a thing that surprises me. Async/await as a concurrency mechanism is so well integrated into the language that you just… write regular code. You don’t need a bunch of concurrency primitive APIs. ~In this case, the `chunked` function from swift algorithms would be perfect.~ Edit: actually hold on, that wouldn’t be quite the same. It would wait for batches of three, but won’t run a 4th task until all 3 of the first 3 tasks were done — Alexander, Dec 15 '22 at 02:24
@matt oh I see, that’s cool! Should we de duplicate this question against that one? — Alexander, Dec 15 '22 at 02:27

Ernie Thomason · Answer 1 · 2022-12-15T03:08:20.487

Thank you all for the help! Martin's link was really helpful

Yes, this is basically a duplicate question, it seems.

The below code will start off 3, then as one finishes, the next starts right away so 3 are always going at once (until the end).

func downloadAllFiles() async -> (String) {
    
    var urls = MArray<String>() ; for _ in 1 ... 10 { urls.add("https://google.com") }
    
    await withTaskGroup(of: Void.self) { group in
        for i in 0 ..< urls.count {
            let url = urls[i]
            if i >= 3 { await group.next() }    // max 3 at a time
            
            group.addTask {
                let web = await MWeb.call(url: url)
            }
        }
        await group.waitForAll()
    }
    return "result"
}

I’m glad it helped! I linked your question to the original so readers can follow it for Rutherford reading — Alexander, Dec 15 '22 at 04:01

Semaphore with async in Swift

1 Answers1