In general Bash sources some files on startup; the details depend on how it was invoked. See Bash Startup Files. Solely by reading this documentation one cannot really tell how exactly an SSH server (or your SSH server) invokes bash, so it's not immediately obvious which set of startup files is used.
Usually (and most likely in your case) one gets an interactive login shell and this fragment of the documentation applies:
Invoked as an interactive login shell, or with --login
When Bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The --noprofile option may be used when the shell is started to inhibit this behavior. […]
There is another fragment titled "Invoked by remote shell daemon" but my tests indicate it applies to a situation when one runs a shell command (as opposed to an interactive shell) via SSH.
My hypothesis is you created ~/.bash_profile and/or ~/.bash_login, therefore ~/.profile is no longer "the first one that exists and is readable". Your ~/.profile probably detects Bash and sources ~/.bashrc (verify this) and the coloring is defined in the latter file.
Ask yourself if you need ~/.bash_profile or ~/.bash_login; maybe what you put there could (or should) be somewhere else. Please read Difference between .bashrc and .bash_profile.
If you really need ~/.bash_profile or ~/.bash_login then make it source ~/.profile maybe; or make it conditionally source ~/.bashrc like your ~/.profile supposably does.
What to run and what to source in what order from what file should be your informed decision. I cannot give you more detailed hint because I don't know what you put in ~/.bash_profile or ~/.bash_login nor what you tried to achieve. I only suspect one of these files prevents ~/.profile (and thus ~/.bashrc, indirectly) from being sourced at startup.