NSOpenPanel.runModal + NSAlert.runModal over GCD cause a hang

Question

In my Cocoa application I have some computations done in the background. The background work is runned with DispatchQueue.global(qos: .utility).async. This background task may report errors by showing a modal NSAlert via DispatchQueue.main.async.

Also, in my application user can run NSOpenPanel to open some files (with NSOpenPanel.runModal).

The problem is that if the user opens NSOpenPanel and at the same time the background task shows NSAlert, the application may hang.

• user opens modal NSOpenPanel
• background task opens modal NSAlert atop the NSOpenPanel
• user clicks Close inside NSOpenPanel (it really can access NSOpenPanel despite the more modal NSAlert present).
• both NSAlert and NSOpenPanel being closed and application hangs with main thread blocked inside NSOpenPanel.runModal()
• application will not hang if user first close NSAlert and then NSOpenPanel.

The minimal code sample (the test IBaction is binded as the action for the button in the main window)

import Cocoa

@NSApplicationMain
class AppDelegate: NSObject, NSApplicationDelegate {

    @IBOutlet weak var window: NSWindow!
   
    @IBAction func test(_ sender: Any) {
        //run some work in background
        DispatchQueue.global(qos: .utility).async
        {
            sleep(1) //some work

            //report errors in the main thread.
            DispatchQueue.main.async {
                let alert = NSAlert();
                alert.informativeText = "Close NSOpen panel before this alert to reproduct the hang."
                alert.runModal()
            }
        }
        
        //user want to open a file and opens the open file dialog
        let dlg = NSOpenPanel();
        dlg.runModal();
    }
}

So, what's wrong with this code and why it cause the hang in particular use cases? And how can I prevent such a hang?

Additional note: I discovered, then if I replace dlg.runModal() with NSApp.RunModal(for: dlg) (which is exactly the same as per Apple documentation), this will fix hang in the usecase described above. But it still will close NSAlert automatically right after closing NSOpenPanel. And I still can't understood why this behave as it does.

Update

I updated the code above to include full code of AppDelegate class of minimal reproducible application. To reproduce the case, just create new SwiftApp in XCode, replace AppDelegate code, add button in the main window and wire button's action with test func. I also placed full ready-to-compile project on github: https://github.com/snechaev/hangTest

The code for configuring NSOpenPanel and NSAlert along with their results handling is excluded as such a code does not affect the hang.

Answer 1

I think you are deadlocking the main queue because runModal() blocks the main queue in two places of your code at the same time. If your code is reentrant, that is what you get.

Possible solutions:

1. Please avoid the use of app modal windows and use instead window modal windows a.k.a. sheets. To know how to use NSOpenPanel as a sheet, attach it to the window it pertains to. For an example, please see this answer:

NSOpenPanel as sheet

2. You can set a flag that prevents the user from opening the NSOpenPanel if the alert is being shown, but that is ugly and does not solve any future problems that can cause other deadlocks because most probably your code is reentrant.

Answer 2

I would like to add some details in addition to @jvarela answer and make some resume about the my issue.

1. Looks like there is no way to solve the problem with having the NSPanel/NSAlert as modal windows with blocking the caller thread (with runModal).
2. The non-modal (NSPanel.begin()) or modal non-blocking (NSPanel.beginSheet, NSPanel.beginSheetModal) do not lead the hang, but still lead to to unexpected automatic closing of NSAlert if user try to close NSPanel before NSAlert. Moreover, to use such a non-blocking approach, you will be forced to refactor the whole your codebase to use callbacks/completion handlers instead of blocking operations when using NSPanel.
3. I do not found the reason why the NSPanel not being blocked and continue to receive user input when I show modal NSAlert atop of it. I suspect that is because of security mechanism which run NSPanel in a separate process, but I have no evidences about this. And I still interested about this.
4. For the my current project I decide to leave the blocking way to use of NSPanel, because I have large codebase and it will hard to change it all in moment to use of completion handlers. For particuluar case with NSPanel + NSAlert I just don't allow user to open NSPanel while this particular background work in progress. The user now should wait background work to finish or manually cancel the work to be able to run Open file functionality.