You need to remove the trailing word boundary that requires a letter, digit or _ to immediately follow the ] char. 
sed 's|\<a\[aa]|bbb|g' file
See the online sed demo:
s="say: a[aa] to bbb, not ba[aa]"
sed 's|\<a\[aa]|bbb|g' <<< "$s"
# => say: bbb to bbb, not ba[aa]
You may also require a non-word char with a capturing group and replace with a backreference:
sed -E 's~([^_[:alnum:]]|^)a\[aa]([^_[:alnum:]]|$)~\1bbb\2~g' file
Here, ([^_[:alnum:]]|^) captures any non-word char or start of string into Group 1 and ([^_[:alnum:]]|$) matches and caprures into Group 2 any char other than _, digit or letter, and the \1 and \2 placeholders restore these values in the result. This, however, does not allow consecutive matches, so you may still use \< before a to play it safe: sed -E 's~\<a\[aa]([^_[:alnum:]]|$)~bbb\1~g'. file`.
See this online demo.
To enforce whitespace boundaries you may use
sed -E 's~([[:space:]]|^)a\[aa]([[:space:]]|$)~\1bbb\2~g' file
Or, in your case, just a trailing whitespace boundary seems to be enough:
sed -E 's~\<a\[aa]([[:space:]]|$)~bbb\1~g' file