Windows Batch: string with exclamation mark used as parameter to function not parsed correctly

Question

I have a text file in which I would like to change the commented strings to some other value, or vise versa. Commented strings in the text file begin with an exclamation mark (!). I am using the FindReplace function as mentioned in the following article: Batch script to find and replace a string in text file within a minute for files upto 12 MB

When I use the :FindReplace function with strings containing an !, I suspect that delayedExpansion is trying to interpret the ! as part of a variable, so when FindReplace is called, the entire line is missing in the file. I've tried to escape the ! in the string by using ^ or ^^, and it doesn't seem to work.

setlocal enableDelayedExpansion
REM //bunch of other script
REM . . . [rest of script not shown]
set /p user_name="Enter user name: %=%"
call :FindReplace "username :" "username: %user_name%" tmpfile.cfg

REM  Comments in txt file start with an !
call :FindReplace "!This is a comment" "This is no longer a comment" tmpfile.cfg
exit /b

:FindReplace 
::<findstr> <replstr> <file>
set tmp="%temp%\tmp.txt"
If not exist %temp%\_.vbs call :MakeReplace
for /f "tokens=*" %%a in ('dir "%3" /s /b /a-d /on') do (
  for /f "usebackq" %%b in (`Findstr /mic:"%~1" "%%a"`) do (
    <%%a cscript //nologo %temp%\_.vbs "%~1" "%~2">%tmp%
    if exist %tmp% move /Y %tmp% "%%~dpnxa">nul
  )
)
del %temp%\_.vbs
exit /b


:MakeReplace
>%temp%\_.vbs echo with Wscript
>>%temp%\_.vbs echo set args=.arguments
>>%temp%\_.vbs echo .StdOut.Write _
>>%temp%\_.vbs echo Replace(.StdIn.ReadAll,args(0),args(1),1,-1,1)
>>%temp%\_.vbs echo end with

How can I make this work correctly?

Edit (1): Looks like I need "enableDelayedExpansion" based on the following "if" statement below. If I have the "enableDelayedExpansion" commented out, the "if" block doesn't execute.

IF EXIST "%PROD_PATH%\%OS_VER%\bin\prod.exe" (
    ECHO There appears to be an installation of PROD in %PROD_PATH%
    SET /p overwrite_input="Would you like to overwrite this installation? [y/n] %=%"
    IF "%overwrite_input%" == "n" ( 
        ECHO Installation Terminating...
        ECHO -----------------------------------------------------------------------------------------------
        PAUSE
        EXIT /B
    ) ELSE IF "%overwrite_input%" == "y" ( 
        ECHO Uninstalling existing application...
        call :Uninstall "%PROD_PRE_COPY%"
        ECHO Continuing installation...
        ECHO -----------------------------------------------------------------------------------------------
    ) ELSE ( 
        ECHO Error: Operation Invalid. Please enter 'y' for yes or 'n' for no without quotes
        ECHO Installation Terminating...
        ECHO -----------------------------------------------------------------------------------------------
        PAUSE
        EXIT /B
    )
 )

There is no need to enable delayed expansion in your script; if you still want it though, you can escape `!` with `^^!` usually (e. g., `echo Hello^^!`); wenn transferring a string to a subroutine via `call`, you'll perhaps need something like `^^^!`, or even more `^`s... — aschipfl, Jan 07 '16 at 21:03
@aschipfl - I'm not a batch expert, but there may be code in the script which requires delayed expansion. I do know that if I turn it off, it will cause the script to execute out of order. In any case, I tried ^^^!Hello, which replaced the line in the file with ^. When I tried 4 ( ^^^^!Hello), I got a line of 4 ^. — Jenner, Jan 07 '16 at 22:12
I see... I think the problem is that `call` variables like `%1`, `%2`, etc. are expanded to their values quite early, so using such will most probably always cause trouble; instead of passing arguments to the subroutine you could perhaps define kind of global variables that hold the values you want to pass to `:FindReplace`, say `strFind` and `strRepl`, so you can expand them in the subroutine using delayed expansion, `!strFind!` and `!strRepl!`... — aschipfl, Jan 07 '16 at 22:38

score 3 · Answer 1 · edited May 23 '17 at 11:51

The reason for the strange bahaviour is that (subroutine) arguments like %1, %2, %3, etc. are parsed at a very early state, long before delayed variable expansion is accomplished (actually even before immediate expansion is done).

To overcome this you need to avoid passing strings/values to the subroutine as arguments. I see the following options:

Global Variables

Here you need to assign the strings/values you want the subroutine to process to global variables, before calling the subroutine. These are read in the subroutine with delayed expansion:

setlocal EnableDelayedExpansion

rem ...SKIPPING SOME CODE...

rem Define global variables and read them in subroutine:
set "strFind=!This is a comment"
set "strRepl=This is no longer a comment"
set "fileTmp=tmpfile.cfg"

call :FindReplace

endlocal
exit /b

:FindReplace
set "tmpf=%temp%\tmp.txt"
If not exist %"temp%\_.vbs" call :MakeReplace
for /F "tokens=*" %%a in ('dir /S /B /A:-D /O:N "%fileTmp%"') do (
  for /F "usebackq" %%b in (`Findstr /MIC:"!strFind!" "%%a"`) do (
    <%%a cscript //nologo %temp%\_.vbs "!strFind!" "!strRepl!">"%tmpf%"
    if exist "%tmpf%" move /Y "%tmpf%" "%%~dpnxa">nul
  )
)
del %temp%\_.vbs
exit /b

rem ...SKIPPING SOME CODE...

Passing Variable Names

Here you need to assign the strings/values you want the subroutine to process to variables, before calling the subroutine. The names of the variables are then to be passed over to the subroutine as arguments. The variables are read in the subroutine indirectly, using delayed expansion:

setlocal EnableDelayedExpansion

rem ...SKIPPING SOME CODE...

rem Define variables and pass their names to the subroutine:
set "strFind=!This is a comment"
set "strRepl=This is no longer a comment"
set "fileTmp=tmpfile.cfg"

call :FindReplace strFind strRepl fileTmp

endlocal
exit /b

:FindReplace
set "tmpf=%temp%\tmp.txt"
If not exist %"temp%\_.vbs" call :MakeReplace
for /F "tokens=*" %%a in ('dir /S /B /A:-D /O:N "!%~3!"') do (
  for /F "usebackq" %%b in (`Findstr /MIC:"!%~1!" "%%a"`) do (
    <%%a cscript //nologo %temp%\_.vbs "!%~1!" "!%~2!">"%tmpf%"
    if exist "%tmpf%" move /Y "%tmpf%" "%%~dpnxa">nul
  )
)
del %temp%\_.vbs
exit /b

rem ...SKIPPING SOME CODE...

Note:
To find out how the command prompt cmd parses scripts you may be interested in this great thread.

As FYI, I tried your suggestions, but I'm encountering script issues where script execution is out of order. I'll have to see if there are any internal resources which can review the entire script as a code review. — Jenner, Jan 08 '16 at 17:10

Henrik · Answer 2 · 2016-01-11T22:12:23.460

2

Change your setlocal line to

setlocal DISABLEDELAYEDEXPANSION

and it should work.

To successfully escape ! you must keep in mind that the shell passes every command twice and unescapes once every time. So to echo a ! you must do

echo ^^!

However, if you assign ! to a var and later echo it you need yet another ^ :

set x=this works^^^!
echo %x%

edited Jan 11 '16 at 22:12

answered Jan 07 '16 at 20:01

Henrik

332
3
12

I tried to disable delayed expansion prior to my call and then re-enable after my call to FindReplace. The script then executed out of order and the temp file didn't comment/uncomment the line. – Jenner Jan 07 '16 at 22:04
But you don't need delayed expansion anywhere in your script. – Henrik Jan 07 '16 at 22:07
See my new edit. Looks like I need delayed expansion, or it only works with it. – Jenner Jan 11 '16 at 20:17
@Jenner No, you don't use delayed exp so you don't need it. But you may need setlocal. Try the line I gave you above. By the way: To escape ! follow my instruction above. – Henrik Jan 11 '16 at 22:07

score 2 · Answer 3 · answered Jan 11 '16 at 20:42

REM . . . [rest of script not shown]
SETLOCAL DISABLEDELAYEDEXPANSION
set /p user_name="Enter user name: %=%"
call :FindReplace "username :" "username: %user_name%" tmpfile.cfg

REM  Comments in txt file start with an !
call :FindReplace "!This is a comment" "This is no longer a comment" tmpfile.cfg
ENDLOCAL

The additional setlocal disabledelayedexpansion / endlocal bracket positioned as indicated should solve the problem.

score 0 · Accepted Answer · answered Jan 12 '16 at 19:26

OK, I finally got it working. First, I want to thank Henrik, Magoo, and aschipfl for their suggestions. I had to do a hybrid of the suggestions, which is why I'm providing this answer.

setlocal enableDelayedExpansion
REM //bunch of other script
REM . . . [rest of script not shown]
set /p user_name="Enter user name: %=%"
call :FindReplace "username :" "username: %user_name%" tmpfile.cfg

REM Adding this section, as it may be pertinent to the DelayedExpansion
echo Select your destination:"
echo 1) local
echo 2) Far
set /p menu_selection="selection? %=%"
if "!selection!" == "2" (
    set "advancedOpt=1"
)
REM  Comments in txt file start with an !
call :FindReplace "!This is a comment" "This is no longer a comment" tmpfile.cfg

REM using aschipfl's FindReplaceWithVar, but also need Henrik's DisableDelayedExpansion
setlocal DisableDelayedExpansion
set "varStrFind=!This is a comment"
set "varStrRepl=This is no longer a comment"
set "varFile=tmpfile.cfg"
call :FindReplaceWithVar varStrFind varStrRepl varFile
endlocal

REM for some reason, the script only works when I do another setlocal endlocal, not sure why I can't reassign the variables.
setlocal DisableDelayedExpansion
set "varStrFind=!This is another comment"
set "varStrRepl=This is not another comment"
set "varFile=tmpfile.cfg"
call :FindReplaceWithVar varStrFind varStrRepl varFile
endlocal

REM now back to using delayed expansion
if "!advancedOpt!" == "1" (
    setlocal EnableDelayedExpansion
    set "varStrFind=This option shouldn't be enabled"
    set "varStrRepl=!Now I'm not enabled."
    set "varFile=tmpfile.cfg"
    call :FindReplaceWithVar varStrFind varStrRepl varFile
    endlocal
)

exit /b

REM I still have the original :FindReplace method, but need to also use the method below.

REM This is aschipfl's modification.  Need to EnableDelayedExpansion for the method
:FindReplaceWithVar
setlocal Enable
set "tmpf=%temp%\tmp.txt"
If not exist %"temp%\_.vbs" call :MakeReplace
for /F "tokens=*" %%a in ('dir /S /B /A:-D /O:N "!%~3!"') do (
  for /F "usebackq" %%b in (`Findstr /MIC:"!%~1!" "%%a"`) do (
    <%%a cscript //nologo %temp%\_.vbs "!%~1!" "!%~2!">"%tmpf%"
    if exist "%tmpf%" move /Y "%tmpf%" "%%~dpnxa">nul
  )
)
del %temp%\_.vbs
endlocal
exit /b


:MakeReplace
>%temp%\_.vbs echo with Wscript
>>%temp%\_.vbs echo set args=.arguments
>>%temp%\_.vbs echo .StdOut.Write _
>>%temp%\_.vbs echo Replace(.StdIn.ReadAll,args(0),args(1),1,-1,1)
>>%temp%\_.vbs echo end with

So, as mentioned in the script, I needed to enable DelayedExpansion in aschipfl's FindReplace methods.

I also need to surround each of the strings which contain a "!" with a setlocal DisableDelayedExpansion ... endlocal. I thought I could continue with the DisableDelayedExpansion with the "!This is another comment" from the preceding block, but the varStrFind and other variables wouldn't set properly or would execute the script out of order.

So, at least this is working, and I'm happy with that.

Windows Batch: string with exclamation mark used as parameter to function not parsed correctly

4 Answers4

Global Variables

Passing Variable Names