The underlying API for creating a hotspot/tether/WiFi access point only supports 2.4Ghz and 5GHz wifi bands in the current build of Windows 11 (22621):
Fields
Auto 0 Specifies that the WiFi adapter is free to choose any band per internal logic.
FiveGigahertz 2 Specifies that the WiFi adapter should use only the 5 GHz band.
TwoPointFourGigahertz 1 Specifies that the WiFi adapter should use only the 2.4 GHz band.
However, the current insider preview build of Windows 11 (29541) has added a new SixGigahertz value. I don't have the preview build installed, so I'm not sure whether there's a matching option in the graphical interface yet. If you can't wait, there's a starter powershell example to launch the wifi tethering here, though you'll have to feed it the Band settings.
Either way, the feature should be coming Soon™, at least for Windows 11. I couldn't find the same docs for Windows 10 builds, but I didn't search very far