While I haven't been able to reproduce your issue, let me share what I know (so far), since it got too long for a comment/chat. I just set up a fresh 20.04 WSL2 (Windows 11) distribution to attempt to reproduce your issue, but it works for me -- I'm able to launch into WSLg with sudo nautilus --new-window.
This Ask Ubuntu answer on a question referencing a similar error seems to indicate a conflict in trying to run more than one Nautilus instance, but I'm not able to reproduce that way either.
A couple of notes:
First, and most importantly, this probably isn't an issue with all GUI apps. I would recommend trying:
sudo apt install xterm
sudo xterm
I expect that this will either work for you (with a root shell in xterm), or help us identify your root issue (no pun intended; just a happy accident).
You might also try wsl --terminate <distribution_name> and trying Nautilus-as-root in a freshly booted instance.
And, there's also a chance that sudo -E nautilus --new-window might give you a different result, but I think that's unlikely based on my testing. Still, worth a shot.
Ultimately, one important thing to realize is that running Gnome apps under WSLg is not as simple as running most other GUI apps. Gnome apps have a high-level of dependency on many other inter-connected features, such as:
I thought I would be able to track your particular issue to one of these, but I haven't been able to connect-the-dots to reproduce it on my system yet.
It's typically possible to overcome Gnome issues on WSL, but it definitely requires more effort than other apps.
For instance, even as your regular user, you'll probably get an Operation not supported error in Nautilus if you attempt to access the "Trash" feature. This can be overcome with:
dbus-launch nautilus
... which will launch Nautilus inside a D-bus user session, enabling gvfs support through D-bus.
Which raises a final thought -- You might also try:
sudo dbus-launch nautilus