TLDR
result = sorted(lst, key=lambda s: [(c.lower(), c.isupper()) for c in s])
You can transform each string to a list of tuples, one per character. A tuple for a character c takes a form (c.lower(), c.isupper()). The usual list comparison gives your desired sort.
lst = ["a", "aa", "aaa", "A", "AA", "AAA", "b", "bb", "bbb", "B", "BB", "BBB"]
lsts = [[(c.lower(), c.isupper()) for c in s] for s in lst]
# [[('a', False)],
# [('a', False), ('a', False)],
# [('a', False), ('a', False), ('a', False)],
# [('a', True)],
# [('a', True), ('a', True)],
# [('a', True), ('a', True), ('a', True)],
# [('b', False)],
# [('b', False), ('b', False)],
# [('b', False), ('b', False), ('b', False)],
# [('b', True)],
# [('b', True), ('b', True)],
# [('b', True), ('b', True), ('b', True)]]
res = ["".join(c.upper() if u else c for c, u in ls) for ls in lsts]
Recovering the result:
['a', 'aa', 'aaa', 'A', 'AA', 'AAA', 'b', 'bb', 'bbb', 'B', 'BB', 'BBB']
Note that there are many distinct ways to order mixed-case elements consistent with the OPs original example. This approach is the only reasonable sort that I can think of which arises from an anti-symmetric order relation. In particular, this sort admits no equivalent elements that are not equal.
For example, ['aAa', 'aaA'] and ['aaA', 'aAa'] will lead to the same output of ['aaA', 'aAa'].