How to search & replace arbitrary literal strings in sed and awk (and perl)

Question

Say we have some arbitrary literals in a file that we need to replace with some other literal.

Normally, we'd just reach for sed(1) or awk(1) and code something like:

sed "s/$target/$replacement/g" file.txt

But what if the $target and/or $replacement could contain characters that are sensitive to sed(1) such as regular expressions. You could escape them but suppose you don't know what they are - they are arbitrary, ok? You'd need to code up something to escape all possible sensitive characters - including the '/' separator. eg

t=$( echo "$target" | sed 's/\./\\./g; s/\*/\\*/g; s/\[/\\[/g; ...' ) # arghhh!

That's pretty awkward for such a simple problem.

perl(1) has \Q ... \E quotes but even that can't cope with the '/' separator in $target.

perl -pe "s/\Q$target\E/$replacement/g" file.txt

I just posted an answer!! So my real question is, "is there a better way to do literal replacements in sed/awk/perl?"

If not, I'll leave this here in case it comes in useful.

Answer 1

The quotemeta, which implements \Q, absolutely does what you ask for

all ASCII characters not matching /[A-Za-z_0-9]/ will be preceded by a backslash

Since this is presumably in a shell script, the problem is really of how and when shell variables get interpolated and so what the Perl program ends up seeing.

The best way is to avoid working out that interpolation mess and instead properly pass those shell variables to the Perl one-liner. This can be done in several ways; see this post for details.

Either pass the shell variables simply as arguments

#!/bin/bash

# define $target

perl -pe"BEGIN { $patt = shift }; s{\Q$patt}{$replacement}g" "$target" file.txt

where the needed arguments are removed from @ARGV and utilized in a BEGIN block, so before the runtime; then file.txt gets processed. There is no need for \E in the regex here.

Or, use the -s switch, which enables command-line switches for the program

# define $target, etc

perl -s -pe"s{\Q$patt}{$replacement}g" -- -patt="$target" file.txt

The -- is needed to mark the start of arguments, and switches must come before filenames.

Finally, you can also export the shell variables, which can then be used in the Perl script via %ENV; but in general I'd rather recommend either of the above two approaches.

---

A full example

#!/bin/bash
# Last modified: 2019 Jan 06 (22:15)

target="/{"
replacement="&"

echo "Replace $target with $replacement"

perl -wE'
    BEGIN { $p = shift; $r = shift }; 
    $_=q(ah/{yes); s/\Q$p/$r/; say
' "$target" "$replacement"

This prints

Replace /{ with &
ah&yes

where I've used characters mentioned in a comment.

The other way

#!/bin/bash
# Last modified: 2019 Jan 06 (22:05)

target="/{"
replacement="&"

echo "Replace $target with $replacement"

perl -s -wE'$_ = q(ah/{yes); s/\Q$patt/$repl/; say' \
    -- -patt="$target" -repl="$replacement"

where code is broken over lines for readability here (and thus needs the \). Same printout.

Answer 2

Me again!

Here's a simpler way using xxd(1):

t=$( echo -n "$target" | xxd -p | tr -d '\n')
r=$( echo -n "$replacement" | xxd -p | tr -d '\n')
xxd -p file.txt | sed "s/$t/$r/g" | xxd -p -r

... so we're hex-encoding the original text with xxd(1) and doing search-replacement using hex-encoded search strings. Finally we hex-decode the result.

EDIT: I forgot to remove \n from the xxd output (| tr -d '\n') so that patterns can span the 60-column output of xxd. Of course, this relies on GNU sed's ability to operate on very long lines (limited only by memory).

EDIT: this also works on multi-line targets eg

target=$'foo\nbar' replacement=$'bar\nfoo'

Answer 3

With awk you could do it like this:

awk -v t="$target" -v r="$replacement" '{gsub(t,r)}' file

The above expects t to be a regular expression, to use it a string you can use

awk -v t="$target" -v r="$replacement" '{while(i=index($0,t)){$0 = substr($0,1,i-1) r substr($0,i+length(t))} print}' file

Inspired from this post

Note that this won't work properly if the replacement string contains the target. The above link has solutions for that too.

Answer 4

This is an enhancement of wef’s answer.

We can remove the issue of the special meaning of various special characters and strings (^, ., [, *, $, \(, \), \{, \}, \+, \?, &, \1, …, whatever, and the / delimiter) by removing the special characters.  Specifically, we can convert everything to hex; then we have only 0-9 and a-f to deal with.  This example demonstrates the principle:

$ echo -n '3.14' | xxd
0000000: 332e 3134                                3.14

$ echo -n 'pi'   | xxd
0000000: 7069                                     pi

$ echo '3.14 is a transcendental number.  3614 is an integer.' | xxd
0000000: 332e 3134 2069 7320 6120 7472 616e 7363  3.14 is a transc
0000010: 656e 6465 6e74 616c 206e 756d 6265 722e  endental number.
0000020: 2020 3336 3134 2069 7320 616e 2069 6e74    3614 is an int
0000030: 6567 6572 2e0a                           eger..

$ echo "3.14 is a transcendental number.  3614 is an integer." | xxd -p \
                                                       | sed 's/332e3134/7069/g' | xxd -p -r
pi is a transcendental number.  3614 is an integer.

whereas, of course, sed 's/3.14/pi/g' would also change 3614.

The above is a slight oversimplification; it doesn’t account for boundaries.  Consider this (somewhat contrived) example:

$ echo -n 'E' | xxd
0000000: 45                                       E

$ echo -n 'g' | xxd
0000000: 67                                       g

$ echo '$Q Eak!' | xxd
0000000: 2451 2045 616b 210a                      $Q Eak!.

$ echo '$Q Eak!' | xxd -p | sed 's/45/67/g' | xxd -p -r
&q gak!

Because $ (24) and Q (51) combine to form 2451, the s/45/67/g command rips it apart from the inside.  It changes 2451 to 2671, which is &q (26 + 71).  We can prevent that by separating the bytes of data in the search text, the replacement text and the file with spaces.  Here’s a stylized solution:

encode() {
        xxd -p    -- "$@" | sed 's/../& /g' | tr -d '\n'
}
decode() {
        xxd -p -r -- "$@"
}
left=$( printf '%s' "$search"      | encode)
right=$(printf '%s' "$replacement" | encode)
encode file.txt | sed "s/$left/$right/g" | decode

I defined an encode function because I used that functionality three times, and then I defined decode for symmetry.  If you don’t want to define a decode function, just change the last line to

encode file.txt | sed "s/$left/$right/g" | xxd -p –r

Note that the encode function triples the size of the data (text) in the file, and then sends it through sed as a single line — without even having a newline at the end.  GNU sed seems to be able to handle this; other versions might not be able to.

As an added bonus, this solution handles multi-line search and replace (in other words, search and replacement strings that contain newline(s)).

Answer 5

I can explain why this doesn't work:

perl(1) has \Q ... \E quotes but even that can't cope with the '/' separator in $target.

The reason is because the \Q and \E (quotemeta) escapes are processed after the regex is parsed, and a regex is not parsed unless there are valid pattern delimiters defining a regex.

As an example, here's an attempt to replace the string /etc/ in /etc/hosts by using a variable in a string passed to perl:

$target="/etc/";
perl -pe "s/\Q$target\E/XXX/" <<<"/etc/hosts";

After the shell expands the variable in the string, perl receives the command s/\Q/etc/\E/XXX/ which is not a valid regex because it doesn't contain three pattern delimiters (perl sees five delimiters, i.e., s/…/…/…/…/). Therefore, the \Q and \E are never even executed.

The solution, as @zdim suggested, is to pass the variables to perl in a way that they are included in the regex after the regex is parsed, such as like this:

perl -s -pe 's/\Q$target\E/XXX/ig' -- -target="/etc/" <<<"/etc/123"

Answer 6

awk escaping is not all that complex either :

on the searching regex, just these 2 suffices to escape any and all awk variants - simply "cage" all of them, with additional escaping performed for just the circumflex/caret, and backslash itself :

-- technically you don't need to escape space at all - sometimes i like using it for marking an unambiguous anchoring point for the character instead of letting awk be too agile about how it handles spaces and tabs. swap the space for "!" inside the regex if u like

  jot -s '' -c  - 32 126 | 

  mawk 'gsub("[[-\440{-~:-@ -/]", "[&]") \       

                  gsub(/\\|\^/, "\\\\&")^_' FS='^$' RS='^$'  

• \440 is (`) - i'm just not a fan of having those exposed in my code
  

|

  [ ][!]["][#][$][%][&]['][(][)][*][+] [,] [-][.] [/]   # re-aligned for 
  0123456789                    [:][;] [<] [=][>] [?]   # readability
  [@]ABCDEFGHIJKLMNOPQRSTUVWXYZ    [[][\\] []][\^][_]
  [`]abcdefghijklmnopqrstuvwxyz    [{] [|] [}][~]

as for replacement, only literal "&" needs to be escaped via

gsub(target_regex, "&")         # nothing escaped

      matched text

gsub(target_regex, "\\&")       # 2 backslashes

      literal "&"

gsub("[[:punct:]]", "\\\\&")    # 4 backslashes 

  \!\"\#\$\%\&\'\(\)\*\+\,\-\.\/\:\;\<\=\>\?\@\[\\\]\^\_\`\{\|\}\~

—- (personally prefer using square-brackets i.e. char classes as an escaping mechanism than having backslash galore)

gsub("[[:punct:]]", "\\\\\\&")   # 6 backslashes

  \&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&\&

Use 6-backslashes only if you're planning to feed this output further down to another gsub()/match() function call