And the peer-discovery functionality that I've built works by simply sharing the IP address of connected nodes along with their announced port for incoming connections.
There are a few issues with this. Assuming the server is outside the NAT while all clients are inside, this means the server will only learn the NAT gateway's address, which won't work either for the server or even for any other client, even if that other client is inside. (That is, client B must be connecting to client A's internal address, not to their common NAT gateway address.)
As far as I know, at least some modern peer-discovery protocols (see e.g. WebRTC) have the client explicitly share all of its IP addresses with the discovery server – e.g. even if you connected via IPv4, the server (and other peers) also learn your IPv6 address, your other IPv4 addresses (internal, external, perhaps internal #2), etc.
IPv6 mostly avoids this issue because there is (usually!) no address translation; however, it is fairly common for IPv6 hosts to have two sets of addresses – a global address and a local address – so it is still useful to report both.
More generally, assuming the two peers are on different sides of a NAT (or even worse, both behind separate NATs), the "announced port for incoming connections" will generally not work through a typical NAT, unless the client specifically used either NAT-PMP or UPnP IGD to request an inbound port mapping from its local NAT gateway (and that also only works if there's just one layer of NAT, no lazy "stacked routers" setup and no ISP-level CGNAT). In case port mapping is not available, you'll need a "NAT traversal" mechanism.
When I connect through IPv4 there is no match between the two reported addresses
Yes, so that implies NAT (at least one layer thereof). Practically all of the time, NAT is "1:Many" NAT where multiple hosts share the same external IP address, in which case new inbound connections from 'outside' to that address will go nowhere without an explicit mapping for some TCP port to some internal address (technically they go to the NAT gateway itself, which will just refuse them), whereas new inbound connections from 'inside' will go nowhere even with an explicit mapping (NAT-hairpin is a whole headache on its own).
So if you want connections from 'inside' to work (e.g. two adjacent PCs in the same office), then the clients have to self-report their addresses instead of relying on "what the server sees". I'm nowhere near an expert in this area, but webrtc ice candidates might be a place to start research.
Meanwhile, for connections from 'outside', your options are a) have the clients attempt to reserve a port mapping with NAT-PMP and/or UPnP IGD – and, again, self-report not just the reserved port they received from the gateway (which may well be different from the port that the client is listening on) but also the "external" address as well; and b) use NAT traversal mechanisms such as STUN, which often do not work with TCP.
