The holy grail would be such that I can plug the Yubikey in any computer, download/configure a client and connect to the VPN server.
The issue with this plan is that configuring a VPN client generally requires Administrator rights. You can do that on your own PCs but not so much as a guest in random places (where the smart card would make the most sense). It also exposes your network to the unknown computer. Anywhere a portable key would be useful, SSH or HTTPS will work better.
So the question is if you know of a VPN client/server that allows RSA certificate authentication?
Many do – it's very common in enterprise networks. IKEv2 (IPsec), L2TP/IPsec, OpenVPN, OpenConnect (Cisco's AnyConnect), are just a few examples. Quite possibly Windows built-in SSTP – given that it's TLS-based – and technically even PPTP (uncommon but the Windows PPTP client can be convinced to do EAP-PEAP so it likely could do cert-based EAP as well).
However, as you noted, it is not enough for software to merely "use RSA keys" since the whole point of your Yubikey is that the software cannot access the actual RSA keys; the software has to specifically know how to request a signature from the card.
And the problem is that practically no software except GnuPG interfaces with OpenPGP cards (nor can verify trust of OpenPGP certificates, card or not) – everything else works either with PKCS#11 or Windows CAPI as the standard interfaces to smart-card contents, and generally with X.509 format certificates (the kind used for TLS) as the carrier for RSA public keys.
(Yes, GnuPG can emulate ssh-agent so that OpenSSH can use it for raw RSA, but hardly any software aside from SSH clients supports using an ssh-agent as a generic interface, either – though it's possible in theory.)
In short, it's not realistic to have OpenPGP as your sole PKI infrastructure. At least I don't think it's going to happen on Windows; it's maybe possible on Linux to use a PKCS#11-to-GnuPG adapter but will be a massive headache even there, given how 1) it was really buggy the last time I tried it, and 2) the VPN client is normally a system service and therefore cannot really depend on a user-level GnuPG 'scdaemon' process.
So instead you will have to start an X.509 PKI using the PIV (CCID) module that your Yubikeys also have. The Yubikey 4 has twelve slots for RSA2048 or P256/P384 certificates (the older Yubikey NEO has four and each is specific-purpose):
Latest tools:
ykman piv keys generate -a RSA2048 9a tmp.pub
ykman piv certificates generate -s "Example" -d 3650 9a tmp.pub
ykman piv certificates export 9a tmp.crt
Slightly older:
ykman piv generate-key -a RSA2048 9a tmp.pub
ykman piv generate-certificate -s "Example" -d 3650 9a tmp.pub
(Here 9a refers to a PIV certificate slot. These are specific to PIV-style cards, not PKCS#11 in general, and are similar to auth/enc/sign slots in OpenPGP cards – though Yubikey 4's implementation makes them more generic.)
The (emulated) PIV card can then be used with most things that support PKCS#11 on Linux, or anything that supports CAPI on Windows:
Windows has a PIV card driver built in (due to PIV being the US DoD standard), so any software that uses CryptoAPI (CAPI) or CNG will automatically recognize the certificate.
PuTTY-CAC/PageantCAC is one example, and of course the Windows built-in IKE clients (both IKEv2 and L2TP/IPsec) are another.
Some software like 'OpenSSH for Windows' wants a PKCS#11 module even on Windows, in which case you'll need the DLLs from yubico-piv-tool:
ssh -I libykcs11.dll root@somehost
OpenSSH on Linux:
ssh-keygen -D libykcs11.so > yubikey_id_rsa.pub
ssh -I libykcs11.so root@server
opensc-pkcs11.so also works, as PIV is not Yubikey-specific.
OpenVPN on Linux:
openvpn --show-pkcs11-ids should load the right module via p11-kit.
strongSwan for IPsec: https://docs.strongswan.org/docs/latest/plugins/pkcs11.html
Similarly, web browsers can authenticate to websites using a TLS "client certificate" – both Firefox and Chrome will load them from CAPI on Windows, and can be told to use PKCS#11 on Linux.
You can then extend this to using one Yubikey as an X.509 CA that signs e.g. device-bound certificates (both Windows and Linux can store an X.509 RSA certificate on your computer's TPM – which is also accessible via PKCS#11 and CAPI – and you can carry around a "roaming" certificate in a second Yubikey), although it does need some plan for deploying revocation data for it to be really secure, but it sounds like not having one is your current state anyway.