(The following explanation turned out to be WRONG though it explained everything. See the UPDATE as follows.)
- # !xxx
 - This works as expected because - !is in the comment.
 
- echo # !xxx
 - This also works as expected because - !is also in the comment.
 
- echo `true # !xxx`
 - This also works because - !is still in the comment, though it's in the- `...`context.
 
- echo `# !xxx`
 - Why doesn't this work? - I guess there's a little bug when Bash interprets the - `...`part. In- `...`, Bash always assumes (wrongly) the first WORD is  a COMMAND name so it does not think- !is in a comment and so history expansion is triggered. That's to say,- echo `# !xxx`is just like- echo `COMMAND !xxx`.
 
- echo `# # !xxx`
 - Why does this work? - As explained in #4, the first - #is parsed as a COMMAND so it's just like- echo `COMMAND # !xxx`so now- !is in the comment.
 
- echo `## !xxx`
 - This double hash does not work either. - As explained in #4 and #5, here - ##is the first WORD and it's parsed as the COMMAND name so it's also like- echo `COMMAND !xxx`.
 
Note that, in the `...` context, the bug is only in the first round syntax parser. That's to say, even though Bash initially parses the # as a COMMAND name, it does not really run it as a command which is named #.
UPDATE 2020-03-04
The above explanation turned out to be WRONG though it explained everything. Please see the discussion in bug-bash mailing list.
I'd quote Chet's explanation here for easy reference:
> $ set -H
> $ true `# !xxx`
> bash: !xxx`: event not found
Well, the history comment character (#) is not found at the start of a
word (here # is part of the word `#), so the rest of the line is processed for history expansion.
$ true `# # !xxx`
The history comment character is found at the start of a word (here the 2nd # itself is a word) and history
expansion skips the rest of the line.
Readline history expansion knows very little about shell syntax; in
particular, it doesn't know backquotes. It never has.