This is a more complex question because it's not always Windows that makes this decision, so it's not always Windows that must be forced to prefer one vs the other (short of disabling IPv6 entirely, that is).
In Windows, when a hostname lookup returns multiple addresses, the OS uses an "address label" or "prefix policy" mechanism to sort them. For example, the "6to4" and "site-local" IPv6 address ranges are prioritized below IPv4, but the rest of IPv6 is above IPv4. (Unless the only address that your computer has is also a 6to4 one. The "matching labels" part adds some complexity.)
On Windows you can use netsh or Get-NetPrefixPolicy to see the precedence list (on Linux the file /etc/gai.conf has the same function).
netsh interface ipv6 show prefixpolicies
To prioritize IPv4 over IPv6, you can add a higher precedence to the ::ffff:0:0/96 prefix (which represents IPv6-mapped IPv4 addresses):
netsh interface ipv6 add prefixpolicy prefix=::ffff:0:0/96 precedence=100 label=4
It will now have a higher priority than the catch-all ::/0 entry representing the entirety of IPv6 & IPv4. (The original precedence for this prefix was 35, whereas ::/0 has precedence 40.)
However, not all programs use the OS-provided order. Browsers, in particular, use a so-called "Happy Eyeballs" algorithm (of which there are a few versions), simultaneously starting an IPv4 and IPv6 connection and using whichever one is the first to establish. Depending on version, the connections may be started at the same time, or IPv6 might be given a head start, often ignoring the "prefix policy" order in either case.
If the program implements its own address ordering, you may be out of luck.
If you only need IPv6 for specific destinations, then there is a heavy duty option of removing the "default route" for IPv6, and statically configuring routes only for those destinations. (This assumes that your gateway IP address won't change, which is hopefully the case with most routers.)
netsh interface ipv6 set interface "Wi-Fi" ignoredefaultroutes=enable
route -6 del ::/0
route -6 -p add 2001:db8::/48 fe80::<gateway_address> if <wifi_interface_index>
(But note that the absence of a default route may cause some programs, such as Chrome, to completely avoid using IPv6 – even for destinations that you do have specific routes for.)
Finally, you could just disable IPv6 for all your interfaces, as there are no popular services so far that'd be IPv6-only; so unless you need IPv6 for a specific destination, you mainly just lose the benefit of inbound connections. (This is assuming that both of your ISPs provide IPv4 in the normal way, and not via NAT64 or similar mechanisms.)
