It depends on the context. Your RFCs list different limits because they talk about different things – the SMTP limit of 255 is specifically for the domain-part of local@domain email addresses and has little, if anything, to do with the limit set by the DNS protocol.
(This is in part because SMTP predates DNS by years – it was designed when HOSTS.TXT was still in use on the ARPANET (see RFC 810/811); and it used to exist in a world where email was transported across 10 different networks so that an SMTP email address domain might have not been an Internet domain at all.)
Prerequisites
To start with an important detail:
However, some people claim it is 254 because of the possible presence of a dot at the end, although there is no mention of such dot in the RFCs
There is. RFC 1034 section 3.1:
When a user needs to type a domain name, the length of each label is
omitted and the labels are separated by dots ("."). Since a complete
domain name ends with the root label, this leads to a printed form which
ends in a dot. We use this property to distinguish between:
a character string which represents a complete domain name
(often called "absolute"). For example, "poneria.ISI.EDU."
a character string that represents the starting labels of a
domain name which is incomplete, and should be completed by
local software using knowledge of the local domain (often
called "relative"). For example, "poneria" used in the
ISI.EDU domain.
Relative names are either taken relative to a well known origin, or to a
list of domains used as a search list. Relative names appear mostly at
the user interface, where their interpretation varies from
implementation to implementation, and in master files, where they are
relative to a single origin domain name. The most common interpretation
uses the root "." as either the single origin or as one of the members
of the search list, so a multi-label relative name is often one where
the trailing dot has been omitted to save typing.
So per specification, www.google.com is not the full domain name – www.google.com. is – and the trailing dot is merely "omitted to save typing", but is very much valid and has a specific purpose. You add a trailing dot to tell the system that a local domain should not be appended.
As a practical example, ICANN used to allow top-level domains to have "host" records. For example, the tk. TLD had a website at http://tk./ – which had to be written with the trailing dot, otherwise your system would look up tk.dlink.lan or whatever.
The RFC uses a single-label example, and many assume that all systems treat any multi-component domain name as already absolute, but that is not always the case – Windows, in particular, does not limit itself like that.
Instead, if a DNS search suffix is configured in Windows, it will be applied to any non-absolute name even if it has multiple components. For example, if corp.example.com is your system's domain, then www.google.com will be first attempted as www.google.com.corp.example.com. and only then as www.google.com..
In this situation, you would explicitly add the trailing dot to your input to prevent the OS from automatically suffixing it with the local domain.
The other RFC that you were reading – RFC 1035 – does not talk about the "trailing dot" because it mostly concerns itself with the network protocol, while the user-facing details are in RFC 1034.
But the text representation is not where the actual length limit comes from.
The actual limits
The real limit is "255 octets or less" when applied to the domain name as it is represented in DNS network ('wire') format, which doesn't use 'dot' separators – it indeed uses a length-prefix byte – and the equivalent of the "trailing dot" is not optional.
In this format each string is prefixed with a length byte, and the entire list is terminated by a 0-byte string. To quote RFC 1035:
Domain names in messages are expressed in terms of a sequence of labels.
Each label is represented as a one octet length field followed by that
number of octets. Since every domain name ends with the null label of
the root, a domain name is terminated by a length byte of zero. The
high order two bits of every length octet must be zero, and the
remaining six bits of the length field limit the label to 63 octets or
less.
For example, www.google.com would become:
14 characters: www.google.com
Interpreted as: [3] "www"
[6] "google"
[3] "com"
[0] "" ("null label of the root")
16 bytes: <0x03> w w w <0x06> g o o g l e <0x03> c o m <0x00>
(The trailing 0x00 byte here has nothing to do with C strings; in a way it's the complete opposite – the format is much more like a series of Pascal-strings – but really it is not related to any programming language at all. It is specifically the way domain names are transferred in the network protocol.)
So according to the same RFC, the limit for domain names in this specific form is 255 bytes:
To simplify implementations, the total length of a domain name (i.e.,
label octets and label length octets) is restricted to 255 octets or
less.
It happens that the binary (network) representation is almost always¹ two bytes longer than the commonly used text representation (i.e. without the trailing dot). So if a domain name is limited to 255 octets on the wire, it is therefore limited to 253 ASCII characters in text form (plus optional trailing dot as the 254th character).
In other words, all three numbers are correct in their own right – but they are often "claimed" without any context to interpret them with; 255 is correct for the packet format, 254 is correct for the ASCII format with trailing dot (absolute name), and 253 is correct for the "conventional" ASCII format without trailing dot.
¹ However, there is an edge case – a literal dot can be written as \., which is one octet in binary form but represented as two in text – but that's extremely rare in practice; the only time you'll see this is when looking at the SOA "responsible person" field, which might have an email address written as fred\.foobar\.jr.example.com.
22 characters: ab\.cd\.ef.example.com
Interpreted: [8] "ab.cd.ef" (a single label with dots!)
[7] "example"
[3] "com"
[0] "" ("null label of the root")
22 bytes: <x08> a b . c d . e f <x07> e x a m p l e <x03> c o m <x00>
If you're writing a DNS library, then you must account for this (by always measuring the length of the wire format, not the text format), but in most other situations it's safe to ignore it. This was once part of a grand plan to use DNS for routing individual mailboxes (with dedicated DNS rrtypes such as 'MG' for mailing-list expansion) which never came to pass, and survives only within SOA rname values.
Since your module accepts domain names in ASCII format, it should:
Allow at most one trailing dot – best to either always add or always strip it before continuing validation (for consistency).
Allow at most 254 bytes if trailing dot is present, or at most 253 bytes if it's removed (simple method)... or convert the name to 'wire format' and then allow that to be at most 255 bytes (accurate method).
Merge most of the "top level domain validity" checks into the regular "label" checks, as the basic rules are exactly the same for every label no matter if it's first or last or in the middle.
(While there do exist additional rules for TLDs, they aren't set by protocol – they are set by policy by ICANN as the maintainer of the root zone which defines TLDs. So while it's reasonable to have e.g. a "begins with a digit" rule for TLDs, it should be kept separate.)
Have a mode that relaxes the "allowed characters" rule. Domain names in general may contain any character – only domain names that are used as hostnames have stronger restrictions, and those come from them being "used as hostnames" (i.e. for A/AAAA lookup), not from their being "domain names".
At DNS level, non-"hostname" domain names are allowed to contain almost anything – e.g. they commonly contain characters like _ (which are practically mandatory for SRV records; _minecraft._tcp.example.com is a typical example) – so at the very minimum, domain name validation needs to distinguish between "hostname (no underscore)" and "non-hostname (underscore OK)" as different contexts.
