Is a public DNS like 8.8.8.8 is a DNS server or just resolver?
It's just a resolver like those provided by ISPs. It does not change anything about how your domain is managed: the records are still kept in the zonefile in your 'authoritative' servers, and you sign the zone with your own keys.
If my users use public DNS like 8.8.8.8, how can I configure my own DNS TXT records there.
You don't. Since 8.8.8.8 is a resolver, it will follow the chain of delegations (NS records) and will get the records from your own authoritative server. (Exactly the same way as resolving other domains like "google.com"...)
So if the domain is public (bought from a registrar) and if your DNS server is publicly accessible, then nothing will change.
Of course, if the domain isn't delegated – e.g. if it's under some made-up TLD like mycompany.lan – then public resolvers won't be able to see it at all. So don't use made-up domains.
How public DNS works exactly to resolve example.com that uses DNSSEC to authenticate in its DNS server
For regular DNS record retrieval, see above, and see plenty of existing documentation. In short, the resolver follows the chain of NS records from the root zone until it finds your own server.
DNSSEC validation is very similar: in addition to NS records (pointing to the next server), each delegation also has DS records (having the next zone's public key). The resolver or validator again follows the chain of DS records.
