js2010's helpful answer is correct in that the use of Start-Process is incidental to your question and that the behavior is specific to PowerShell's CLI (powershell.exe for Windows PowerShell, and pwsh for PowerShell (Core) 6+):
On Windows[1], there are two layers of evaluation to consider:
Re (a):
As most console programs on Windows do, PowerShell recognizes only " chars. (double quotes) - not also ' (single quotes) - as having syntactic function.[2]
That is, unless " chars. are escaped as \", they are treated as command-line argument delimiters with purely syntactic function, which are therefore removed during parsing.
A quick example: powershell -c "hi!" makes PowerShell strip the " and try to execute hi! as a command; by contrast, powershell -c \"hi!\" escapes and therefore retains the " as part of the command, so that PowerShell executes "hi!", i.e. a string literal that is echoed as such.
Note: If you're calling from cmd.exe, \" escape sequences can situationally break the invocation; for robust workarounds see this answer.
As implied by the above, ' chars. are not removed.
Re (b)
Whatever arguments are the result - an array of possibly "-stripped tokens - are concatenated with a single space between them, and the resulting (single) string is then interpreted as PowerShell code.
To explain this in the context of your example, with Start-Process taken out of the picture:
Note: The following applies to calling from cmd.exe or any context where no shell is involved (including Start-Process and the Windows Run (WinKey-R) dialog). By contrast, PowerShell re-quotes the command line behind the scenes to always use ", if needed.
To put it differently: the following applies to command lines as seen by PowerShell.
Single-quoted command:
# Note: This *would* work for calling ping if run from
# (a) PowerShell itself or (b) from a POSIX-like shell such as Bash.
# However, via cmd.exe or any context where *no* shell is involved,
# notably Start-Process and the Windows Run dialog, it does not.
powershell -Command 'ping google.com'
(a) results in PowerShell finding the following two verbatim arguments: 'ping and google.com'
(b) concatenates these verbatim arguments to form 'ping google.com'[2] and executes that as PowerShell code, therefore outputs the content of this string literal, ping google.com
Double-quoted command:
powershell -Command "ping google.com"
(a) results in PowerShell stripping the syntactic " characters, finding the following, single verbatim argument: ping google.com
(b) then results in this verbatim argument - ping google.com - being executed as PowerShell code, which therefore results in a command invocation, namely of the ping executable with argument google.com.
[1] On Unix-like platforms, the first layer doesn't apply, because programs being invoked only ever see an array of verbatim arguments, not a command line that they themselves must parse into arguments. Not that if you call the PowerShell CLI from a POSIX-like shell such as bash on Unix-like platforms, it is that shell that recognizes single-quotes as string delimiters, and strips them before PowerShell sees them.
[2] Surprisingly, on Windows it is ultimately up to each individual program to interpret the command line, and some do choose to also recognize ' as string delimiters (e.g., Ruby). However, many programs on Windows - including PowerShell itself - are based on the C runtime, which only recognizes ".
[3] As an aside, note that this implies that whitespace normalization is taking place: that is,
powershell -Command 'ping google.com' would equally result in 'ping google.com'.
