Prepare a string in a block of parenthesis variable expansion is disabled to return it to a calling scope that enables expansion (no call allowed)

Question

i have a request for theorical knowledge purpose only in Windows Batch scripting.

Today my question is about preparing the value of a local string loc_str INSIDE A BLOCK OF PARENTHESIS and WITHOUT ENABLING DELAYED EXPANSION, in order to return it to a local scope that enables variable expasion (or eventually to prepare it for an incoming local for loop with in("%loc_str%") if nasty characters in loc_str have been properly escaped before).

We assume that loc_str contains at least one ^ and one ! and do not contain double quotes. Lets consider the following code:

@echo off
setlocal disabledelayedexpansion
set "flag_dde_prev=%flag_dde%" & set "flag_dde=!"
(
    set "loc_str=Hello^^^ planet!!!! ^^Earth^"

    if not defined flag_dde_prev (
        call set "out_str=%%loc_str:^=^^%%"
        set out_str

        call set "out_str=%%out_str:^=^^%%"
        set out_str
    )
)
endlocal & set "str=%out_str%"
set str

As you know the local flags flag_{dde, dde_prev} are used to test the type of the calling and current scopes. Typically they're both defined at the beginning of a block setlocal..endlocal. The flag_dde is equal to ! if the current scope disables variable expansion, or is undefined if the current scope enables it. The local value of flag_dde_prev is the inherited value of flag_dde in the calling scope.

What we must do here is to escape all ^ and ! to prepare the return of out_str with %..% to the calling scope when this latter enable variable expansion (ie when flag_dde_prev is defined). Two substitutions ^=^^ then !=^! with a simple set would be enough, but being inside a block (..) forces us to use the call set statement. Unfortunatly the caret ^ is not replaced in the same way they are in a scope that enables variable expansion.

Precisely, the first substitution ^=^^ doubles even sequences of carets, but does not double odd ones (it doubles them minus one). Then, the second substitution !=^! replace each ! with ^^!.

To sump up,

loc_str=Hello^^^ planet!!!! ^^Earth^               ;init
out_str=Hello^^^^^ planet!!!! ^^^^Earth^           ;1st substitution ^=^^
out_str=Hello^^^^^ planet^^!^^!^^!^^! ^^^^Earth^   :2nd substitution !=^!

^=^^ seems to be reversible with ^^=^ if done BEFORE !=^! only. If ^^=^ is done after !=^! then sequences of ^ not preceeding ! are modified only. If ^^=^ is done after !=^! i didn't find a way to replace the sequences of ^^ before each ". It behaves like a sequence ^^ before ! is the atomic unreplacable one, longer caret sequences can be replaced but i couldn't obtain ^! whatever i tried.

The same problem is quite easy to solve when variable expansion is enabled, even with nasty strings containing quotes by replacing "" by " at first (jeb already talked about this in another thread). For example if locs_tr doesn't contain quotes, the substitutions would be the following:

set "out_str=!loc_str:^=^^^^!"              ;1st substitution, multiplicates the number of ^ by 4 as expected
call set "out_str=%%out_str:^!=^^^!%%" !    ;2nd substitution, replaces each ! by ^^!
set "out_str=!out_str:^^=^!"                ;3rd substitution required, replaces each ^^ by ^ to divide by 2 the total number of ^

So my question is simple: Is there a way using substitution IN THE EXACT SAME CONTEXT (ie no call) to obtain the desired prepared output value for out_str, so it can be returned safely to a scope that enables delayed expansion ?

The prepared output value should double the number of ^ and add ^ before each ! ie:

out_str=Hello^^^^^^ planet^!^!^!^! ^^^^Earth^^

Note that's my question is for theorical knowledge purpose ONLY. Indeed calling a label to do the two substitutions with a simple set is the reasonable way to proceed.

Before I even attempt to work out what your specific question may be, I have to ask, what is the already existing value of `%flag_dde%`? Also, is `flag_dde_pre` supposed to be the same as the same as `flag_dde_prev`, i.e. a typo, or is that also something you have omitted from your [mcve]? — Compo, Jun 20 '22 at 19:13
Do you insist on `WITHOUT ENABLING DELAYED EXPANSION`? Because it's easier to temporarily enable it to solve your problem — jeb, Jun 20 '22 at 19:17
Hi compo, flag_dde is always set to "!" at the beginning of each setlocal..endlocal scopes so its value is either undefined or "!". For flag_dde_pre is that a typing mistake, i corrected to flag_dde_prev. — Lionel Blanc-guilhon, Jun 20 '22 at 19:34
Hi jeb, thanks for your interest as well. I'm not against a solution with expansion anabled temporarily if ever you have one, would it look like the same than the substitutions i did at the end ? — Lionel Blanc-guilhon, Jun 20 '22 at 19:36

jeb · Answer 1 · 2022-06-20T20:18:00.723

1

Your assumptions are wrong!

What we must do here is to escape all ^ and ! to prepare the return ...
Two substitutions ^=^^ then !=^! with a simple set would be enough ...

The problem is here the fact that caret folding is different if there is an exclamation mark in the line or not!

setlocal EnableDelayedExpansion
echo "one caret^"
echo "no caret^ but a bang^!"
echo "caret ^^ and a bang^!"
echo one caret ^^
echo no caret^^ but a bang^^!
echo caret ^^^^ and a bang^^!

call and carets and exclamation marks are really funny.

Because call always double all carets before any other rule, but you don't see it with quotes, because in the next parser phase the doubled carets are folded to single carets again.

Parse flow for a single line like

Starting with:
   call echo my caret^^^^ "caret^^" and a bang !

1. Caret escaping outside quotes
   call echo my caret^^ "caret^^" and a bang !

2. delayed expansion caret escaping (outside and **inside** quotes), the bang itself is lost here
   call echo my caret^ "caret^" and a bang

3. The `CALL` caret doubling
   call echo my caret^^ "caret^^" and a bang

4. Caret escaping outside quotes
   call echo my caret^ "caret^^" and a bang

The problem itself was solved by dbenham and me a few years ago.
See SO: Macro to preserve variables when leaving setlocal scope
or SO: Make an environment variable survive ENDLOCAL.

In your case it's much simpler, because you don't support quotes nor line feeds in the variable.
And you know, that you leave the current scope and will return into a scope with delayed expansion enabled, this simplifies the problem, too.

@echo off

setlocal EnableDelayedExpansion

setlocal disabledelayedexpansion
(
    set "loc_str=Hello^^^ planet!!!! ^^Earth^"
    set "loc_str"

    setlocal EnableDelayedExpansion
    set "out_str=!loc_str:^=^^^^!"              # 1st substitution, multiply all carets by four
    set "out_str"
    call set "out_str=%%out_str:^!=^^^!%%" !  # 2nd substitution, replace all exclamation marks to two carets with exclam
    set "out_str"
    set "out_str=!out_str:^^=^!"              # 3rd substitution reduce all carets by factor of two
    set "out_str"
    for /F "delims=" %%V in ("!out_str!") do (
        endlocal 
        endlocal
        set "str=%%V" !  # The trailing exclamation mark is intentionally, but will not part of the result
    )
    
)

echo str = '!str!'

edited Jun 20 '22 at 20:18

answered Jun 20 '22 at 19:23

jeb

78,592
17
171
225

They can be a source of headaches as well (smile). Did the rules you write above apply samewise when expansion is enabled or not ? I don't master enough how a line is parsed, i should dig into the full thread posted here to avoid as much as "try and guess". Do you mean that the substitutions ^=^^ and !=^! are what must be done to protect the output string if this latter does to contain any double quotes ? – Lionel Blanc-guilhon Jun 20 '22 at 19:46
Thanks a lot jeb, i will read your two links tomorow (it's alreadt late in Farnce) and will tell you if i understood all or if i need explains. The general macro in your first link will be usefull, because a general solution whatever the types of current/calling scopes would be perfect if i also did a mistake when the current scope enable expansion. And you're right, no quotes and no line feed simplifies the problem. – Lionel Blanc-guilhon Jun 20 '22 at 20:12
@LionelBlanc-guilhon I added a solution. Btw. Avoid your comments like `;3rd substitution required, replaces each ^^ by ^ to divide by 2 the total number of ^` because they **modify** your code! Better avoid special characters in the *inline* comments. In Germany it's nearly the same time than in France, also a bit late :-) – jeb Jun 20 '22 at 20:14
Thanks for the remark, i'm quite new here so you're right to give me good habbits^^ ! I still have old reflexes from assembly langage with ; when i write pseudo code or pseudo output (smile). I couldn't resist to read the page about your %endlocal% macro, at quick first reading i got all except the ^!=""e^! substitution. I have to execute the macro to get why it's done, i presume it has a link with replacing ! by ^! properly ? I dig into tomorow, and will post my questions on the macro page if i feel i need. – Lionel Blanc-guilhon Jun 20 '22 at 21:31
However the performance impact of one call set per variable must also be considered, that's why i didn't use any macro to return an undetermined number of variables for my launcher. This number can be quite high sometimes (especially during the boot phase but also game run one), so some functions "know" they return to a DDE scope necessarily to optimize the output survival process by skipping output protection (output values are returned like in your 2nd link). – Lionel Blanc-guilhon Jun 20 '22 at 21:31
Do you think the ordered substitution with SET only !var:^=^^^^! then !var:""="! then "%var!=^!!!%" ! then !var:""="! are equivalent to the 4 ones done in the macro ? I typically use this substitution sequence from EDE to EDE for a determined number of output variables, and it seems to work pretty well even with the nasty string with quotes you defined in your thread about replacing !. Have a good night in Germany jeb, catch you tomorow to tell you if i got all about the !^=^^e^! . :) – Lionel Blanc-guilhon Jun 20 '22 at 21:32
Hi, i didn't execute your macro yet, instead i started to dig a bi about the folding rules about ^ with/without quotes and bangs. I did read again how the parsing rules you posted in your first link, and the more i was experimenting the more i realized that i didn't insist enough by the past to understand how the parser work. Thus my knowledge about this topic is quite parcellar at the moment, and drives me to confusion in some situations. – Lionel Blanc-guilhon Jun 21 '22 at 15:23
The more i read all again, the more it seems that i also have question about how the parsing rules of the command `SET "variable=value"" when value contains at least 1 quote, and even worst when it contains characters like & |. For example, why `set "var=bub "& bob"` and `set var=bub "& bob` are not equivalent ? Seems like SET doesn't follow the parsing rules, because in first case & shouldn't be considered as special regarding the rule in PHASE 2 about ". Should i open a new thread to go on there for now, then go back to post here here when things seem to me clearer ? – Lionel Blanc-guilhon Jun 21 '22 at 15:27
@LionelBlanc-guilhon The `set "var=bub" & bob"` results into an error, because the `&` is outside of quotes and therefore it splits the line into two commands `set "var=bub"` and `bob"` – jeb Jun 22 '22 at 07:14
But in this case why set var=bub "& bob defines var to "bub" and doesn't result into the same error than the quoted notation ? I did the false assumption (again !) that the interpreter removed surrounded quotes first in set "var=bub" & bob" to assign `bub" & bob`to var. Apparently the interpreter doesn't proceed like this, do you know why such differences ? – Lionel Blanc-guilhon Jun 22 '22 at 19:38

Lionel Blanc-guilhon · Answer 2 · 2022-06-22T11:04:22.790

Ok jeb i just executed your macro. The code below can be copied and run. If i understood properly, the general principle is the following (correct me if i didn't).

@echo off
setlocal enableextensions disabledelayedexpansion
set "_(echo=echo: & echo"
set "_(set=echo: & set"

:: Initialize the DDE flags of the current/previous scopes.
set "flag_dde_prev=" & set "flag_dde=!"

:: Base string ; one of your nasty strings defined in another post ;p
set "str_base=caret ^ bang ! and some (other ) <>&| %% \"

setlocal enabledelayedexpansion
set "flag_dde_prev=!flag_dde!" & set "flag_dde=!"

  setlocal enabledelayedexpansion
  set "flag_dde_prev=!flag_dde!" & set "flag_dde=!"

  :: Nasty local string to be returned ; still one of yours !
  set "loc_str_nasty=!str_base! and also quoted stuff "!str_base!""
  set loc_str & echo:

  if defined flag_dde_prev (
    set "out_str=!loc_str_nasty!"
  ) else (
    rem STEP1: Each " is replaced by ""q so everything is consider outside quotes
    set "out_str=!loc_str_nasty:"=""q!"
    set out_str

    rem STEP2: Each ^ is replaced with ^^
    set "out_str=!out_str:^=^^!"
    set out_str

    rem STEP3: Each ! is replaced by ""e! 
    rem WARN A simple `set` must be avoided because of special characters.
    call set "out_str=%%out_str:^!=""e^!%%"
    set out_str

    rem STEP4/5: Initial " and ^! were marked differently with letters q and e so we can restore them by replacing ""q and ""e! by " and ! respectively
    set "out_str=!out_str:""e=^!"
    set out_str
    set "out_str=!out_str:""q="!"
    set out_str
  )
    
  rem Return the value of out_str to the calling DDE/EDE scope by using a safe `for`.
  for /F "delims=" %%@ in ("!out_str!") do endlocal & set "str_pro=%%@"
    
  :: Echo safely the protected string str_pro in the current DDE/EDE scope
  %_(set% str_pro

  if defined flag_dde (
    for /F "tokens=1* delims==" %%v in ('set str_pro') do echo %%str_pro%%=%%w
  ) else (
    %_(echo% ^^^!str_pro^^^!=!str_pro!
  )

endlocal

Thanks for your remarks, indeed i talked too fast there is no problem at all with ""q and ""e in the input value of loc_str. You're very right about avoiding special characters with rem, i totally forgot. I did typing mistake also by assigning flag_dde_prev, it must be done with !..! inside an EDE scope of course. I struggle with the code editor here, i'm not used to such stuffs for now (yes i know i sound like a cavern man lol).

I edited my previous post, the protection block (..) is skipped when returning to a DDE scope (i forgot also to assign out_str when flag_dde_prev when defined to !). The returned protected string to DDE contains nasty characters so can't be echoed with %..%, so i used a for to echo it.

Is there a way to deal with " and &|<> to protect a such nasty loc_str to echo it in the calling DDE scope with a brutal echo %..% or call set %..%, or at the contrary is an undirect echoing can't be avoided ?

By undirect i mean without a slow echo only ie with a for like above, or with a very fast set str_pro only by deleting the leading string "str_pro=" by adding a dynamic leading string of backspace in the out_str value.

OUTPUT Run twice, first when the calling scope is DDE, second when it's EDE (written as EDE in the code above).

The macro you based on was a simple minded one, later I build code that can also handle linefeeds/carriage returns. But I changed completely the replacement logic for it to work. See the code from dbenham for it [exit function and preserve variable over endlocal barrie](https://www.dostips.com/forum/viewtopic.php?f=3&t=6496&p=41929#p41929) and [return over an endlocal barrier with a macro](https://www.dostips.com/forum/viewtopic.php?t=4827) — jeb, Jun 22 '22 at 07:42
I 'm reading the new version of your macro, i think i don't get what needs to be protected when returning to DDE. I certainly do the false asumption (again !) that returning from EDE to DDE is safe by using a for and a loop variable %%, because i see that you DO protect "safeReturn_Dis" as much as "safeReturn_Ena" before returning it. So maybe can you give me a nasty string example that could crash the code i posted above ? It would help me to have a concreete example why building a safe return to DDE is mandatory before returning it with for. Thanks in advance for your lights — Lionel Blanc-guilhon, Jun 22 '22 at 13:47
Some simple failing strings `===equals removed` or `Line1\nLine2` where `\n` should be a real linefeed — jeb, Jun 22 '22 at 18:03
The common way of echo in DDE is to use temporarily EDE :-) Your `FOR /F "delims==" .. ('set str_pro')` is slow because for the `set str_pro` a new cmd instance will be created and it's prone to errors, because it would output all variables beginning with str_pro, like str_prog. It strip leading equal signs and strings beginnig with `;` are completely removed — jeb, Jun 22 '22 at 18:08
I can't find a string that doesn't contain any " CR LF and that contains at least one = that crashes a `for /F "delims" %%@ in ("!out_str!") do set "out_str=%%~@"`, can you give me an explicit one ? :) — Lionel Blanc-guilhon, Jun 24 '22 at 11:26
Let's continue in [chat](https://chat.stackoverflow.com/rooms/245883/https-stackoverflow-com-q-72691677-463115) — jeb, Jun 24 '22 at 13:36

Prepare a string in a block of parenthesis variable expansion is disabled to return it to a calling scope that enables expansion (no call allowed)

2 Answers2