Verbatim string "Hello must ultimately be passed as \"Hello to rg ("\"Hello" would work too). That is, the verbatim " char. must be \-escaped:
From cmd.exe:
rg \^"Hello
^, cmd.exe's escape character, ensures that the " is treated verbatim and is removed by cmd.exe before calling rg.
Note that ^ isn't strictly necessary here, but it prevents the " from being considered the start of a double-quoted argument, which could make a difference if there were additional arguments.
From PowerShell:
rg \`"Hello
`, PowerShell's escape character, ensures that the " is treated verbatim and is removed by PowerShell before calling rg.
Arguably, the explicit \-escaping shouldn't be necessary, because it is the duty of a shell to properly pass arguments to target executables after the user has satisfied the shell's own escaping requirements (escaping the verbatim " with ^ in cmd.exe, and with ` in PowerShell).
In the context of PowerShell, this problematic behavior is summarized in this answer.
Note that in PowerShell this extra escaping is only needed if you call external programs; it isn't needed PowerShell-internally - such as when you call Select-String, as shown in js2010's answer.