Cmd and exclamation marks - Part II

Question

I'm really wondering why my string replacement procedure works when parsing text files containing any special characters including exclamation marks. I expected that delayed variable expansion would switch off special meaning of ampersand, percent sign etc. but will fail instead for exclamation marks...

Code:

@echo on & setlocal ENABLEEXTENSIONS

set "InFile=%~1"
set "OutFile=%~2"
set "Replace=%~3"

CALL :ParseCue "%%InFile%%" "%%OutFile%%" "%%Replace%%"

endlocal &GOTO:EOF

:ParseCue
@echo on & setlocal ENABLEEXTENSIONS DISABLEDELAYEDEXPANSION
set "FileToParse=%~1"
set "OutputFile=%~2"
set "NewExtension=%~3"
for /F "usebackq tokens=* delims=" %%a in ("%FileToParse%") DO (
  set "line=%%a"
  @echo on & setlocal ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
  set "line=!line:.wav=%NewExtension%!"
  echo(!line!>>"%OutputFile%"
  endlocal
)
endlocal &GOTO:EOF

InputFile.txt:

This a test for parsing lines with special characters
Rock & Roll.wav
Rock & Roll!.wav
Special | < > ~ \ ²³ { [ ] } ! " ´ ' ` üäö @ ; : € $ % & / ( ) = ? chars.wav

Command line syntax:

D:\Users\Public\Batch\YAET>parse.bat "InputFile.txt" "OutputFile.txt" ".flac"

OutputFile.txt:

This a test for parsing lines with special characters
Rock & Roll.flac
Rock & Roll!.flac
Special | < > ~ \ ²³ { [ ] } ! " ´ ' ` üäö @ ; : € $ % & / ( ) = ? chars.flac

---

EDIT / Supplement:

After 1 1/2 years I had to use this code snippet again. See two additional examples handling poison chars. First one with temporary enabled delayed expansion again (see Ansgars answer), second one using CALL. Both will parse path and name of non-empty files in and below current directory, but without trailing drive letter and path to current dir.

Example #1 (Enclosing double quotes in set "File=!File..." and echo "!FILE!">>... are not required):

@echo off & setlocal ENABLEEXTENSIONS DISABLEDELAYEDEXPANSION
del NonEmptyFiles.txt >NUL 2>&1
echo Searching non-empty files in and below current directory ...
for /f "tokens=*" %%I in ('dir /s /b /a:-D') do (
    if not %%~zI==0 (
        set "File=%%I"
        setlocal ENABLEDELAYEDEXPANSION
        set "File=!File:%cd%\=!"
        echo "!File!">> NonEmptyFiles.txt
        endlocal
        )
    )
echo Done. See NonEmptyFiles.txt.
endlocal &goto:EOF

Example #2 (slower, enclosing double quotes required):

@echo off & setlocal ENABLEEXTENSIONS DISABLEDELAYEDEXPANSION
del NonEmptyFiles.txt >NUL 2>&1
echo Searching non-empty files in and below current directory ...
for /f "tokens=*" %%i in ('dir /s /b /a:-D') do (
    if not %%~zi==0 (
        set "File=%%i"
        call set "File=%%File:%cd%\=%%"
        call echo "%%File%%">> NonEmptyFiles.txt
        )
    )
echo Done. See NonEmptyFiles.txt.
endlocal &goto:EOF

Files and folders for testing:

D:\Martin\Any & Path>dir /s /b /a:-D
D:\Martin\Any & Path\Hello! World!.txt
D:\Martin\Any & Path\Rock & Roll\!File! !!File!!.txt
D:\Martin\Any & Path\Rock & Roll\%File% %%File%% %%I.txt
D:\Martin\Any & Path\Rock & Roll\Poison! !§$%&()=`´'_;,.-#+´^ßöäüÖÄÜ°^^#.txt
D:\Martin\Any & Path\Rock & Roll\SizeZero.txt

Output:

D:\Martin\Any & Path>stringinforloop.bat
Searching non-empty files in and below current directory ...
See NonEmptyFiles.txt. Done.

D:\Martin\Any & Path>type NonEmptyFiles.txt
"Hello! World!.txt"
"Rock & Roll\!File! !!File!!.txt"
"Rock & Roll\%File% %%File%% %%I.txt"
"Rock & Roll\Poison! !§$%&()=`´'_;,.-#+´^ßöäüÖÄÜ°^^#.txt"

Enjoy batching! Martin

Answer 1

That's because you enable delayed expansion after set "line=%%a":

set "line=%%a"
@echo on & setlocal ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
set "line=!line:.wav=%NewExtension%!"

If you enable delayed expansion before assigning %%a:

@echo on & setlocal ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
set "line=%%a"
set "line=!line:.wav=%NewExtension%!"

you'll get

Special | < > ~ \ ²³ { [ ] } € $ % & / ( ) = ? chars.flac

instead of

Special | < > ~ \ ²³ { [ ] } ! " ´ ' ` üäö @ ; : € $ % & / ( ) = ? chars.flac

---

Edit: Delayed expansion controls when variables in a statement are expanded. The critical statement in your script is the line

set "line=%%a"

which assigns the value of the loop variable %%a to the variable line. With delayed expansion disabled the literal value of %%a is assigned, because the script interpreter cannot expand %%a at parse time. However, when you enable delayed expansion, bang-variables are expanded at execution time, so the interpreter looks at the value of %%a and expands any !whatever! in it before the result is assigned to the variable line.

Perhaps it'll become clearer with an example. If you add a line

%foo% !foo!

to the input file and define the variable in the script:

@echo on & setlocal ENABLEEXTENSIONS

set "foo=bar"
set "InFile=%~1"
...

When you enable delayed expansion after set "line=%%a" neither %foo% nor !foo! are expanded before %%a is assigned to the variable line (the interpreter doesn't see the value of %%a before execution time), so you get this output:

%foo% !foo!

When you enable delayed expansion before set "line=%%a" the interpreter will expand bang-variables before assigning the result to the variable line, so you get this output:

%foo% bar

%foo% would only be expanded at parse time, at which point the interpreter cannot see the actual value of %%a, so %foo% remains a literal %foo% here.

Further assignments like set "line=!line:.wav=%NewExtension%!" don't affect bangs or percent signs inside a variable, because expansion isn't transitive, i.e. it translates !line! to %foo% bar (or %foo% !foo!) and then stops.

You can force the expansion of (percent-)variables inside variables with the call command, though. A command call set "line=!line!" first expands to call set "line=%foo% bar" in the current context, and then call evaluates set "line=%foo% bar" in a new context where %foo% is expanded to bar as well, so the variable line is assigned the value bar bar.

---

Just as a side note: your code is way too complicated. You'd get the exact same results with this:

set "FileToParse=%~1"
set "OutputFile=%~2"
set "NewExtension=%~3"
for /F "usebackq tokens=* delims=" %%a in ("%FileToParse%") DO (
  set "line=%%a"
  @setlocal ENABLEDELAYEDEXPANSION
  set "line=!line:.wav=%NewExtension%!"
  echo(!line!>>"%OutputFile%"
  endlocal
)