LIKE supports pattern matching using _ for any single character and % for any sequence of characters so this:
SELECT 'thomas' LIKE '%(h|x)%'
doesn't work because LIKE doesn't understand (...) for grouping or | for alternation, those are just literal characters in a LIKE pattern.
SIMILAR TO supports _ and % the same as LIKE but adds grouping with (...), alternation with |, and a few other things so this:
SELECT 'thomas' SIMILAR TO '%(h|x)%'
works as expected.
~* uses POSIX regexes so (...) is for grouping and | is for alternation but % is just a percent sign; that means that this:
SELECT 'thomas' ~* '%(h|x)%'
is looking for an h or x surrounded by percent signs and doesn't work they way you expect it to.
Your ~* version will work if you use a proper regex like:
SELECT 'thomas' ~* '(h|x)' -- alternation
SELECT 'thomas' ~* 'h|x'   -- alternation without an unnecessary grouping
SELECT 'thomas' ~* '[hx]'  -- or a character class
The documentation linked to above covers all of this.