It really depends on your routers, what they run, and how much control you have over them. You have not specified what types of routers they are and the OS on them. VPN would have been one way...
I've setup something like that overlaying existing network connecting L2 over L3 between two sites using L2TPv3, which is essentially a Layer 2 extension over Layer 3 networks. This was not done over the public Internet, but the idea was similar since the underlying network was from a different department of the same organization.
To do so I used VyOS on my boxes on each side and setting up a L2TPv3 tunnel. It can also be encrypted if you add IPsec to it, which in essence would act like a VPN. This allowed me to carry multiple VLANs between the two sides, so it was transparent to the clients on either side.
If your routers support L2TPv3 you can look at their docs how to do it. You can make it work if you can afford Linux routers running VyOS or simply flashing your routers with DD-WRT or similar to change their firmware so you you have a better control over your routers. If you are using something off-the-shelf home routers with their default firmware you are probably out of luck unless you are willing to replace their firmware with an open one.
This was a sample config to enable simple L2TPv3 tunnel on VyOS:
Setup bridge and a VLAN subinterface on both routers:
configure
set interfaces bridge br2 aging '300'
set interfaces bridge br2 hello-time '2'
set interfaces bridge br2 max-age '20'
set interfaces bridge br2 priority '1'
set interfaces bridge br2 stp 'false'
set interfaces ethernet eth1 vif 111 bridge-group bridge 'br2'
set interfaces ethernet eth1 vif 111 description 'VLAN 111'
sh
commit
save
Then on Router A:
Link the bridge interface setup earlier to one end of the tunnel:
set interfaces l2tpv3 l2tpeth2 bridge-group bridge 'br2'
set interfaces l2tpv3 l2tpeth2 destination-port '10111'
set interfaces l2tpv3 l2tpeth2 encapsulation 'ip'
set interfaces l2tpv3 l2tpeth2 local-ip 'public.ip.of.routerA'
set interfaces l2tpv3 l2tpeth2 mtu '1500'
set interfaces l2tpv3 l2tpeth2 peer-session-id '111'
set interfaces l2tpv3 l2tpeth2 peer-tunnel-id '111'
set interfaces l2tpv3 l2tpeth2 remote-ip 'public.ip.of.routerB'
set interfaces l2tpv3 l2tpeth2 session-id '111'
set interfaces l2tpv3 l2tpeth2 source-port '10111'
set interfaces l2tpv3 l2tpeth2 tunnel-id '111'
sh
commit
save
Router B config, essentially the reverse of the above:
set interfaces l2tpv3 l2tpeth2 bridge-group bridge 'br2'
set interfaces l2tpv3 l2tpeth2 destination-port '10111'
set interfaces l2tpv3 l2tpeth2 encapsulation 'ip'
set interfaces l2tpv3 l2tpeth2 local-ip 'public.ip.of.routerB'
set interfaces l2tpv3 l2tpeth2 mtu '1500'
set interfaces l2tpv3 l2tpeth2 peer-session-id '111'
set interfaces l2tpv3 l2tpeth2 peer-tunnel-id '111'
set interfaces l2tpv3 l2tpeth2 remote-ip 'public.ip.of.routerA'
set interfaces l2tpv3 l2tpeth2 session-id '111'
set interfaces l2tpv3 l2tpeth2 source-port '10111'
set interfaces l2tpv3 l2tpeth2 tunnel-id '111'
sh
commit
save
Then monitor the traffic between the two if you setup a session like ping, or whatever:
show interfaces
show interfaces detail
monitor traffic interface eth1.111
To setup an IPSec tunnel over the same, an additional config is needed, as illustrated in their documentation.
The example port 10111 should be allowed on both routers in the above examples. You may need (D)NAT between your public and private IPs of your routers.