Shell special chars blowing up regexp(s) in all: sed, (g)awk, and perl

Question

I would like to change the PS1 in my ~/.bashrc programmatically with any of the the most popular regexp tools { sed, awk or perl }. However, I have problems with special characters. Note that in the default ~/.bashrc there are three PS1 prompt variables set under different conditions, and that they are all different. I want to change only two of them, by commenting out the original PS1 line and under it modifying what used to be the original.

For illustration let me show you the segment of .bashrc before and after the modification to PS1 variable. (BTW, to increase readability I use my custom concatenation markings "join" (\\j\) and "space" (\\s\) ), where the former means the two parts are joined without any white-space char between them, but the latter allows spaces between the two.

Before:

if [ "$color_prompt" = yes ]; then 
  PS1='${debian_chroot:+($debian_chroot)}                          \\j\
    \[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ ' 
else 
  PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\n\$> ' 
fi 
  unset color_prompt force_color_prompt
If this is an xterm set the title to user@host:dir
case "$TERM" in 
  xterm|rxvt) 
    PS1="[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a]$PS1" 
  ;; 
  *) 
  ;; 
  esac

After:

if [ "$color_prompt" = yes ]; then 
#    PS1='${debian_chroot:+($debian_chroot)}                         \\j\ 
    \[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ ' 
    PS1='${debian_chroot:+($debian_chroot)}                          \\j\ 
    \[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\n\$> ' 
  else 
#    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ ' 
    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\n\$> ' 
  fi 
  unset color_prompt force_color_prompt
If this is an xterm set the title to user@host:dir
case "$TERM" in 
  xterm|rxvt) 
    PS1="[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a]$PS1" 
  ;; 
  *) 
  ;; 
  esac

I got the following working:

For those of you, who wish to try this out, let me show you what I got working, and what I've also learned is the problem with sed, and where awk and perl are better, though after playing with them, i.e., awk and perl for a day, I actually got only sed to do less than a half of what I wanted!

In particular pay attention to concatenation trick with a single-quote inside a single-quoted string replaced with {{ ' " ' " ' }}: Namely:

                           internal                    internal
                         single-quote                single-quote
                             vvvvv                       vvvvv
  aString=' this is unquoted '"'"' this is single-quoted '"'"' '
  echo "[$aString]"
  [ this is unquoted ' this is single-quoted ' ]

Internal single-quote is actually two single quotes and a string that happens to be a double-quoted single quote {{ ' " ' " ' }}. Which when put together inside $aString, makes that string made up of five concatenated strings:

    ' this is unquoted '  + "'"  + ' this is single-quoted ' + "'" + ' '

Another mine you'll step on in {{ sed }} is that {{ sed }} for some reason does not handle octal representation of <Esc> character (\033) correctly, apparently it'd be fine with hex, namely, {{ \x1b }} or {{ \x1B }}, but that's not what's in the .bashrc file, because the four characters {{ \033 }} are stored literally as ASCII {{ \, 0, 3 and 3 }} though I believe the shell (Bash, Bourne or Perl for that matter) should convert these occurrences into appropriate single character they represent: the <Esc> character in this case. This looks to me like a can of worms! I spent entire weekend testing different permutations of escaped, double escaped, octal, and hex combinations with {{ sed, awk & perl }} and the only thing I got working is the following without colour-codes in {{ sed }}. Adding colour codes in the mix, just kills everything I know and and everything I tried.

This sed code works:

 29 
 30  ps1_2='    PS1='"'"'${debian_chroot:+($debian_chroot)}\\u\@\\h:\\w\\$ '"'"''
 31 ps1_2n='    PS1='"'"'${debian_chroot:+($debian_chroot)}\\u\@\\h:\\w\\n\\$> '"'"'' 
 32 
 33 sed "s/$ps1_2/#&\n$ps1_2n/" ps1-tFILE.txt

Solved

Scott, thank you very much for your prompt reply, that inspired me exactly in the right direction. I solved my problem by handling smaller pieces first and then merging it all back together. Here's how I did it:

  1 #!/usr/bin/env bash 
  2 # $Log$
  3
  4 # -- The following are the contents of the {{ ps1-tFILE.txt }} file: 
  5
  6 #^   PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]     \\j\
                \u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ ' 
  8 #^ else 
  9 #^   PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\n\$> ' 
 10 
 11 P1='    PS1='"'"'${debian_chroot:+($debian_chroot)}' 
 12 P2='\\\[\\033\[01;32m\\]' 
 13 P3='\\u@\\h' 
 14 P4='\\\[\\033\[00m\\]:' 
 15 P5='\\\[\\033\[01;34m\\]\\w' 
 16 P6='\\\[\\033\[00m\\]' 
 17 P7='\\$ '"'"'' 
 18 
 19 srchPatt_wCtrls="\($P1$P2$P3$P4$P5$P6\)\($P7\)" 
 20 replPatt_wCtrls='#&\n\1\\n\\$> '"'"''
 21
 22 # -- For the convenience I also print out all individual parts  \\s\
            of the match and replace patterns. 
 23 echo "P1=$P1" 
 24 echo "P2=$P2" 
 25 echo "P3=$P3" 
 26 echo "P4=$P4" 
 27 echo "P5=$P5" 
 28 echo "P6=$P6" 
 29 echo "P7=$P7" 
 30 echo "srchPatt_wCtrls=$srchPatt_wCtrls" 
 31 echo "replPatt_wCtrls=$replPatt_wCtrls" 
 32 
 33 ps1_o='    PS1='"'"'${debian_chroot:+($debian_chroot)}\\u\@     \\j\
                    \\h:\\w\\$ '"'"'' 
 34 ps1_r='    PS1='"'"'${debian_chroot:+($debian_chroot)}\\u\@     \\j\
                    \\h:\\w\\n\\$> '"'"'' 
 35 
 36 sed -e "s/$srchPatt_wCtrls/$replPatt_wCtrls/"   \ 
 37     -e "s/$ps1_o/#&\n$ps1_r/" ps1-tFILE.txt 
 38 
 39 exit

My test file

For those of you who have trouble with my ASCII text concatenation conventions I'm also including the contents of my test file (ps1-tFILE.txt) in a scrollable window:

$> cat ps1-tFILE.txt 
    PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
else
    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '

Thanks again, it helped me to sleep it over, and get inspirations from Scott:)

Scott - Слава Україні · Answer 1 · 2021-01-19T23:48:15.420

I don’t know where to begin. I might begin with the question you asked, if you had asked one, but I don’t see one.

You say that the sed code you presented works. It doesn’t work for me on the file you provided, for these reasons:

The Before file that you show already has \n before the \$ (in the second assignment — the one without the color codes), but your ps1_2 string contains “…\\w\\$…” (without \n), so it won’t match the \w\n\$ in the file.
Likewise, your Before file already has > after the \$ (again, in the second assignment), but your ps1_2 string ends with “\\$ ” (without >), so it won’t match the \$> in the file.

Additionally, your script uses \\$ (in ps1_2) to match \$ (in .bashrc). That will probably work, in the context of your script. But it might be safer to use \\\$, since \ and $ are both special characters. See more below.

If you change your Before file to say

  if [ "$color_prompt" = yes ]; then
    PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
  else
    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$> '
  fi

(i.e., remove the \n in the second assignment), and change your script to

 ps1_2='    PS1='"'"'${debian_chroot:+($debian_chroot)}\\u@\\h:\\w\\$> '"'"''
ps1_2n='    PS1='"'"'${debian_chroot:+($debian_chroot)}\\u@\\h:\\w\\n\\$> '"'"''
sed "s/$ps1_2/#&\n$ps1_2n/" ...

(i.e., add > to ps1_2), then it will work.

Obviously, if your Before file doesn’t already have > after \$, then don’t add it to ps1_2.

But consider,

The quick brown fax jumps over the lazy dog.

Well, that’s wrong. We can fix it with

s/The quick brown fax jumps over the lazy dog./The quick brown fox jumps over the lazy dog./

(optionally ending the “old” string with \., to be sure), but why, when

s/a/o/

will work?

By the same token, if your objective is “For every line that assigns a value to PS1 that includes \$, comment it out and replace it with an assignment that inserts a \n before the \$.”, then you can do that with

sed 's/\(.*PS1=.*\)\(\\$.*\)/#\1\2\n\1\\n\2/'

This fixes both line 2 and line 4, while leaving line 11 alone (because it doesn’t contain \$). If your objective is “For every line that assigns a value to PS1 that includes \$, comment it out and replace it with an assignment that inserts a \n before the \$ and a > after the \$.”, it’s only a little more complicated:

sed 's/\(.*PS1=.*\)\(\\\$\)\(.*\)/#\1\2\3\n\1\\n\2>\3/'

Note that I had to use \\\$ here. I guess that it’s because, when sed sees …something$.*, it assumes that the $ means a literal <dollar sign> ($), and not the end of the line, because the $ isn’t at the end of the regular expression, but when it sees $…something$$$.*$, it gets confused, because the $ is at the end of a subexpression. This might be a bug in sed.