Why does this happen?
The maintainers of GNU Coreutils's ls have decided so. The program checks whether its output is a terminal and applies different output formatting (colors, special characters, etc).
For example, it also doesn't suppress "non-printable" characters when output is not a terminal, under the assumption that it's going to be processed by a script and not merely looked at. (Not recommended in many cases but valid in some cases, and the program does specifically account for that.) The quoting-by-default is new but having ls automatically disable it for non-tty output is the same as before.
How to keep the surrounding single quotes?
Use --quoting-style=shell if the names are about to be used as input for a sh/bash command.
But if you're planning to use the output as part of a shell script, consider whether ls is necessary at all. For example, far too often for x in $(ls *.c) is misused when for x in *.c would do a better job.