IPsec tunnel established, nmap working but unable to ping

Question

I have this Ubuntu VM on my network which is supposed to concentrate traffic towards a VPN and route it through it. Now, what happens is (although nothing has changed in configurations, except for Ubuntu updates which might have screwed some settings):

• I cannot ping the addresses on the other side which I'm SURE that should respond
• The machines behind the VM can't reach said addresses anymore
• Take one of these addresses, that I can't ping, and run nmap towards it - I correctly get the open ports! Not just that, but it's an http server so I can telnet 80 and ask it for some data and it works.
• If I run tcpdump when pinging, I obviously see ESP packets going out but not coming back

There's no firewall and this is my nftables:

    #!/usr/sbin/nft -f
flush ruleset

table inet filter {
        chain input {
                type filter hook input priority 0;
        }
        chain forward {
                type filter hook forward priority 0;
        }
        chain output {
                type filter hook output priority 0;
        }
}

table ip nat {

        chain postrouting {
                type nat hook postrouting priority srcnat;
                ip daddr xxx.xxx.xxx.xxx snat to yyy.yyy.yyy.yyy
                ip daddr xxx.xxx.xxx.xxx snat to yyy.yyy.yyy.yyy

more destinations here...
        }
}