Select lines between two patterns using variables inside SED command

Question

I'm new to shell scripting. My requirement is to retrieve lines between two pattern, its working fine if I run it from the terminal without using variables inside sed cmd. But the problem arises when I put all those below cmd in a file and tried to execute it.

#!/bin/sh
word="ajp-qdcls2228.us.qdx.com%2F156.30.35.204-8009-34"
upto="2017-01-03 23:00"
fileC=`cat test.log`
output=`echo $fileC | sed -e "n/\$word/$upto/p"`
printf '%s\n' "$output"

If I use the below cmd in the terminal it works fine

sed -n '/ajp-qdcls2228.us.qdx.com%2F156.30.35.204-8009-34/,/2017-01-03 23:00/ p' test.log

Please suggest a workaround.

The command you say works fine would actually fail with false matches given some your input file contents due to lack of anchors and escaping of regexp metacharacters and backreferences. If you **really** want to do this with sed then see https://stackoverflow.com/a/29626460/1745001 for how to do it but of course you'd be far better off just using awk for string matching instead of regexp matching. [edit] your question to show concise, testable sample input and expected output if you'd like help with how to do this the right way. — Ed Morton, Jul 20 '17 at 22:54

randomir · Accepted Answer · 2017-07-20T18:33:14.320

If we put aside for a moment the fact you shouldn't cat a file to a variable and then echo it for sed filtering, the reason why your command is not working is because you're not quoting the file content variable, fileC when echoing. This will munge together multiple whitespace characters and turn them into a single space. So, you're losing newlines from the file, as well as multiple spaces, tabs, etc.

To fix it, you can write:

fileC=$(cat test.log)
output=$(echo "$fileC" | sed -n "/$word/,/$upto/p")

Note the double-quotes around fileC (and a fixed sed expression, similar to your second example). Without the quotes (try echo $fileC), your fileC is expanded (with the default IFS) into a series of words, each being one argument to echo, and echo will just print those words separated with a single space. Additionally, if the file contains some of the globbing characters (like *), those patterns are also expanded. This is a common bash pitfall.

Much better would be to write it like this:

output=$(sed -n "/$word/,/$upto/p" test.log)

And if your patterns include some of the sed metacharacters, you should really escape them before using with sed, like this:

escape() {
    sed 's/[^^]/[&]/g; s/\^/\\^/g' <<<"$1";
}
output=$(sed -n "/$(escape "$word")/,/$(escape "$upto")/ p" test.log)

score 1 · Answer 2 · answered Jul 20 '17 at 22:57

The correct approach will be something like:

word="ajp-qdcls2228.us.qdx.com%2F156.30.35.204-8009-34"
upto="2017-01-03 23:00"
awk -v beg="$word" -v end="$upto" '$0==beg{f=1} f{print; if ($0==end) exit}' file

but until we see your sample input and output we can't know for sure what it is you need to match on (full lines, partial lines, all text on one line, etc.) or what you want to print (include delimiters, exclude one, exclude both, etc.).

Select lines between two patterns using variables inside SED command

2 Answers2