These instructions work on Windows 10. I haven't tested them on any earlier versions of Windows. If they also work on earlier Windows versions, please edit this answer to say so.
In the Bluetooth settings panel, pair the headset with your computer.
Open the Sound control panel.

Right-click on your headset and choose Connect. Wait until Windows connects to your headset and you hear a beep.
- Note: If Windows silently fails to connect, you must first disconnect the headset from your phone.
Set the headset as your default playback device. Don't choose "Set as Default Communications Device". Instead, choose "Set as Default Device".
Note that the Bluetooth Headset Profile was designed for phone calls, not for music playback. It's fine for listening to podcasts; but, if you're listening to music, the sound quality isn't great. If you want better sound quality, buy a pair of Bluetooth headphones with built-in A2DP (music streaming) support.
Enjoy.
I thank DrZoo for making a post elsewhere which helped me figure out how to do this.