The -replace operator:
uses a regex as the search-pattern operand, and therefore requires a verbatim $ to be escaped as \$ in order to be interpreted as such (an unescaped $ is a regex metacharacter representing the end of the input string)
also uses $ as a metacharacter in the substitution operand, namely to refer to the text that the regex captured ($&) or parts thereof (e.g. $1 to refer to what the first capture group ((...)) captured). In this case, escape it as $$.
- Note: Situationally, even an unescaped
$ may work, namely if, in combination with the subsequent characters, if any, it cannot be interpreted as a placeholder such as $& or $1. However, in the interest of robustness it is better to escape.
Therefore, use:
PS> ',"$"20' -replace ',"\$"', ',"$$",'
,"$",20
Taking a step back:
As Abraham Zinala suggests, for verbatim string replacements, the .Replace() string method is sufficient, whose use doesn't necessitate escaping, and which is faster to boot:
PS> ',"$"20'.Replace(',"$"', ',"$",')
,"$",20
Caveat: Unlike -replace, .Replace() is case-sensitive - invariably in Windows PowerShell and by default in PowerShell (Core) 7+.
See this answer for a detailed juxtaposition of -replace and .Replace(), including how to perform escaping for -replace programmatically.
To put it all together, along with an optimization that reads each file in full, with Get-Content -Raw, for much faster processing (to avoid appending a trailing newline, -NoNewLine is used with Set-Content):
$FilePath = "C:\PT\Test"
Get-ChildItem $FilePath -Filter *.TXT | ForEach-Object {
(Get-Content -Raw $_.FullName).Replace(',"$"', ',"$",') |
Set-Content -NoNewLine $_.FullName
}