5

While trying to provide a comprehensive answer to the question Why is FindStr returning not-found, I encountered strange behaviour of code that involves a pipe. This is some code based on the original question (to be executed in a ):

rem // Set variable `vData` to literally contain `...;%main%\Programs\Go\Bin`:
set "vData=...;%%main%%\Programs\Go\Bin"
set "main=C:\Main"

echo/%vData%| findstr /I /C:"%%main%%\\Programs\\Go\\Bin"

This does not return a match, hence nothing is echoed and ErrorLevel becomes set to 1.

Though when I go through the parsing process step by step, I conclude the opposite, I do expect a match and an ErrorLevel of 0, because:

  1. at first, the whole line is parsed and immediate (%) expansion takes place, hence %vData% becomes expanded and %% become replaced by %, resulting in the following command line that is going to be executed:

    echo/...;%main%\Programs\Go\Bin| findstr /I /C:"%main%\\Programs\\Go\\Bin"

  2. each side of the pipe | is executed in its own new cmd instance by cmd /S /D c, both of which run in cmd context (which affects handling of %-expansion), resulting in these parts:

    • left side of the pipe:

      echo/...;C:\Main\Programs\Go\Bin

    • right side of the pipe:

      findstr /I /C:"C:\Main\\Programs\\Go\\Bin"

      (the search string that findstr finally uses is C:\Main\Programs\Go\Bin as the \ is used as escape character even in literal search mode, /C or /L)

As you can see the final search string actually occurs within the echoed string, therefore I expect a match, but the command lines does not return one. So what is going on here, what do I miss?

When I clear the variable main before executing the pipe command line, I get the expected result, which leads me to the conclusion that the variable main becomes not expanded despite my assumption above (note that in cmd context, %main% is kept literally when the variable is empty). Am I right?

It is getting even more confusing: when I put the right side of the pipe in between parentheses, a match is returned, independent on whether or not the variable main is defined:

echo/%vData%| (findstr /I /C:"%%main%%\\Programs\\Go\\Bin")

Can anyone explain this? Could this be connected to the fact that findstr is an external command, opposed to echo? (I mean, break | echo/%vData% expands the value of main as expected if defined...)

aschipfl
  • 33,626
  • 12
  • 54
  • 99
  • This Returns a Match as expected on Windows 7 – Ben Personick Aug 14 '19 at 04:57
  • IE your original code behaves as one would (and you did) expect. `C:\Admin>echo/%vData%| findstr /I /C:"%%main%%\\Programs\\Go\\Bin"` `...;%C:\Main%\Programs\Go\Bin` `C:\Admin>echo/%vData%` `...;%%main%%\Programs\Go\Bin` – Ben Personick Aug 14 '19 at 05:01
  • Could this be something to Do with Win10 and the changes to allow more robust terminal emulation? -- not sure how as I thought Conhost was still the default for cmd, but I am not running win 10, I have some 2012R2/2016/2019 kicking around and will test on a few tomorrow if you haven't by then to at least see which versions are consistent – Ben Personick Aug 14 '19 at 05:05
  • @BenPersonick, I meant to execute the code in a [tag:batch-file] (sorry, I forgot to mention that in the question -- updated), since `%%`-escaping does not work in [tag:cmd]-context. I am using Windows 7... – aschipfl Aug 14 '19 at 08:15
  • This is some really interesting stuff I thought you were mad to use the %% on main unless it was a typo and you used another variable to hold it like with data, I mostly work in the cmd scripts and usually turn them into pastable CMD scripts after the fact, so I am very thrown off now by that, I woudl not have expected the CLI to behave the way you expected the script to behave, and vice verse. Neat find, but for me, the opposite way, it's a funny feeling of a cart before the horse – Ben Personick Aug 14 '19 at 20:19

2 Answers2

4

I simplified your example to:

@echo off

set "main=abc"
break | findstr /c:"111" %%main%%
break | echo findstr /c:"222" %%main%%

The output is:

FINDSTR: %main% kann nicht geöffnet werden.
findstr /c:"222" abc

This proves that using an exe-file in a pipe results in different behaviour than using an internal batch command.
Only for internal commands a new cmd.exe instance will be created.
That's also the cause why the findstr doesn't expand the percent signs.

This confusing line expands, because the parentheses forces a new cmd.exe instance.

break | (findstr /c:"111" %%main%%)

I will modify the explanation at 5.3 Pipes - How does the Windows Command Interpreter (CMD.EXE) parse scripts?

aschipfl
  • 33,626
  • 12
  • 54
  • 99
jeb
  • 78,592
  • 17
  • 171
  • 225
  • 1
    Thank you very much, that explains it! I just verified that a pipe does not initiate a new `cmd` instance for an external command like `findstr`: `type "hugefile.txt" | findstr /C:"text"` creates two new processes (shown in task manager): `cmd /S /D c" type ... "` and `findstr ...` (no `cmd` there)... – aschipfl Aug 14 '19 at 09:45
  • 2
    @aschipfl Thanks for this nice finding, I’m astonished that nobody stumbeled about this before – jeb Aug 14 '19 at 10:15
  • 2
    Yes, this is a very interesting find! I've updated the cmd.exe parsing rules. Interesting that the following only needs one escape: `echo "&" | findstr ^&`. But the parenthesized form requires double escape: `echo "&" | (findstr ^^^&)`. It all makes sense with the updated rules. – dbenham Aug 14 '19 at 16:39
  • 1
    Updating the cmd.exe parsing rules was tricky due to the 30,000 char limit. I had to edit some other sections to a more concise form to make room. Perhaps you could edit your answer to [Why does delayed expansion fail when inside a piped block of code?](https://stackoverflow.com/q/8192318/1012053). – dbenham Aug 14 '19 at 16:49
  • @dbenham Thanks for updating. It's hard for me, I'm on mobile! Your caret example is nice and one more proof. – jeb Aug 14 '19 at 17:18
  • Guys, I am at a loss, how would you not have run into this in the past when piping to anything? If you are saying the CMD behaves differently for internal commands and automatically puts both sides of the pipe into a separate CMD instance, that woudl seemingly only come up when doing something like echoing to date to get it to print the command's default text or where you CALL the command which is as in my example but how have you tackled this need to expand the variable on the right in the past then if NOt CALL (internal command) or Parenthesis which forces the separate evaluation..? – Ben Personick Aug 14 '19 at 19:43
  • Okay I see, I woudl still have used the `CALL` or `()` on the CLI and it would work, but I wouldn't have noticed that it woudl also work without them, that's bizarre, and definitely shows I have a bias towards scripted `CMD`, as I mostly write and call scripts instead of work at the CLI, less so these days, Powershell got me into using the CLI more interactively. – Ben Personick Aug 14 '19 at 20:08
  • 2
    This behavior only applies to pipes. I confirmed that `for /f .... in ('someCommand') do ...` always invokes cmd.exe to execute someCommand, even if someCommand is an external command. One nice outcome of the pipe design - when explicitly executing a command via `cmd /c`, there is not a second implicit `cmd /c`. So something like `break | cmd /v:on /c echo !cmdcmdline!` is efficient. – dbenham Aug 15 '19 at 14:08
  • I just recognised that a pipe also invokes a new `cmd` instance when the command is a batch file; I updated [How does the Windows Command Interpreter (CMD.EXE) parse scripts?](https://stackoverflow.com/a/4095133) as well as @dbenham's [Why does delayed expansion fail when inside a piped block of code?](https://stackoverflow.com/a/8194279) accordingly, I hope this is fine... – aschipfl Aug 21 '19 at 23:23
  • @aschipfl - Thanks. It seems obvious to me that a new `cmd` instance would be needed for a batch script. But it is a good addition to the rule. – dbenham Aug 22 '19 at 00:02
  • I agree, @dbenham, it is obvious (what else, appart from `cmd`, should execute a batch script), but still I had the feeling it is worth to be mentioned. Anyway, should we request to change the 30000 character limit for answers here? ;-) – aschipfl Aug 22 '19 at 10:53
1

In a CMD script changes things.

My Quick and Dirty understanding of the Reason:

The batch is behaving as expected, there is no need to double up the percents on main even on the other side of a pipe, because there is no expansion happening, it is a normal variable.

IE the order of operations you write is correct, but also is happening on each line individually, I think you are just so close to the problem you didn't notice or am I not understanding the issue?

I am not sure if this is clearly stated so I plan to review what I believe is happening step by step:

set "vData=...;%%main%%\Programs\Go\Bin"

Result Stored in Memory: "...;%main%\Programs\Go\Bin"

ECHOing %vData% yields: "...;%main%\Programs\Go\Bin"

Add in Main

set "main=C:\Main"

Result Stored in Memory: "C:\Main"

ECHOing %main% yields: "C:\Main"

ECHOing '%%main%%' yields: "%C:\Main%"

Now that Main is defined lets Echo vData Again:

Result Stored in Memory is still: "...;%main%\Programs\Go\Bin"

ECHOing "%vData%" Yields: "...;%main%\Programs\Go\Bin"

CALL ECHOing "%vData%" Yields: "...;C:\Main\Programs\Go\Bin"

So this side was relying on the expansion but the FindStr side isn't because %main% was already defined so it was expanded on the first pass.

When parsed by the CMD interpreter, I believe the sequence will be as follows:

Given your original Find String:

echo/%vData%| findstr /I /C:"%%main%%\\Programs\\Go\\Bin"

Becomes:

echo/...;%main%\Programs\Go\Bin| findstr /I /C:"%C:\Main%\\Programs\\Go\\Bin"

Becomes:

echo/...;C:\Main\Programs\Go\Bin | (findstr /I /C:"%C:\Main%\\Programs\\Go\\Bin")

Alternatively, When parsed by the CMD interpreter, I believe the sequence will be as follows:

Given the modified Regex String:

echo/%vData%| findstr /I /C:"%main%\\Programs\\Go\\Bin"

Becomes:

echo/...;%main%\Programs\Go\Bin| findstr /I /C:"C:\main\\Programs\\Go\\Bin"

Becomes:

echo/...;C:\Main\Programs\Go\Bin | (findstr /I /C:"C:\Main\\Programs\\Go\\Bin")

If your concern is that you were actually storing %%Main%% in a variable, say the %_Regex% Variable:

Then generally you'd have to either Wrap the statements on the other side of the Pipe in a Parenthesis or Call findStr in my experience.

IE:

@(
  SETLOCAL
  ECHO OFF
)
rem // Set variable `vData` to literally contain `...;%main%\Programs\Go\Bin`:
set "vData=...;%%main%%\Programs\Go\Bin"
rem // Set variable `_Regex` to literally contain `%main%\\Programs\\Go\\Bin`:
SET "_Regex=%%main%%\\Programs\\Go\\Bin"
rem // Set variable `_Regex` to literally contain `C:\Main`:
set "main=C:\Main"

ECHO(&ECHO(Variable Contents After Main Set:&ECHO(
     echo(Normal:   vData = "%vData%"
     echo(Normal:  _Regex = "%_Regex%"
     echo(Normal:    main = "%main%"
CALL echo(CALLed:   vData = "%vData%"
CALL echo(CALLed:  _Regex = "%_Regex%"


ECHO(&ECHO(Testing the Results of The FindString Methods:
ECHO(==============================================&ECHO(
ECHO( Original ^%%^%%main^%%^%% Method:
echo/%vData%| findstr /I /C:"%%main%%\\Programs\\Go\\Bin"
ECHO(==============================================&ECHO(
ECHO( Using the ^%%_Regex^%% Stored Variable:
echo/%vData%| findstr /I /C:"%_Regex%"
ECHO(==============================================&ECHO(
ECHO( Using just ^%%main^%%:
echo/%vData%| findstr /I /C:"%main%\\Programs\\Go\\Bin"
ECHO(==============================================&ECHO(
ECHO( Using the CALL ^%%_Regex^%% Stored Variable:
echo/%vData%| CALL findstr /I /C:"%_Regex%"
ECHO(==============================================&ECHO(
ECHO( Using the (^%%_Regex^%%) Stored Variable:
echo/%vData%| ( findstr /I /C:"%_Regex%" )
ECHO(==============================================&ECHO(

Results:

C:\Admin>C:\Admin\TestFindStr.cmd

Variable Contents After Main Set:

Normal:   vData = "...;%main%\Programs\Go\Bin"
Normal:  _Regex = "%main%\\Programs\\Go\\Bin"
Normal:    main = "C:\Main"
CALLed:   vData = "...;C:\Main\Programs\Go\Bin"
CALLed:  _Regex = "C:\Main\\Programs\\Go\\Bin"

Testing the Results of The FindString Methods:
==============================================

 Original %%main%% Method:
==============================================

 Using the %_Regex% Stored Variable:
==============================================

 Using just %main%:
...;C:\Main\Programs\Go\Bin
==============================================

 Using the CALL %_Regex% Stored Variable:
...;C:\Main\Programs\Go\Bin
==============================================

 Using the (%_Regex%) Stored Variable:
...;C:\Main\Programs\Go\Bin
==============================================

This is what I woudl normally expect given my experience with CMD as well.

Community
  • 1
  • 1
Ben Personick
  • 3,074
  • 1
  • 22
  • 29