Using sed to delete all lines between two matching patterns

Question

I have a file something like:

# ID 1
blah blah
blah blah
$ description 1
blah blah
# ID 2
blah
$ description 2
blah blah
blah blah

How can I use a sed command to delete all lines between the # and $ line? So the result will become:

# ID 1
$ description 1
blah blah
# ID 2
$ description 2
blah blah
blah blah

Can you please kindly give an explanation as well?

score 106 · Accepted Answer · edited Sep 27 '17 at 10:05

106

Use this sed command to achieve that:

sed '/^#/,/^\$/{/^#/!{/^\$/!d}}' file.txt

Mac users (to prevent extra characters at the end of d command error) need to add semicolons before the closing brackets

sed '/^#/,/^\$/{/^#/!{/^\$/!d;};}' file.txt

OUTPUT

# ID 1
$ description 1
blah blah
# ID 2
$ description 2
blah blah
blah blah

Explanation:

/^#/,/^\$/ will match all the text between lines starting with # to lines starting with $. ^ is used for start of line character. $ is a special character so needs to be escaped.
/^#/! means do following if start of line is not #
/^$/! means do following if start of line is not $
d means delete

So overall it is first matching all the lines from ^# to ^\$ then from those matched lines finding lines that don't match ^# and don't match ^\$ and deleting them using d.

edited Sep 27 '17 at 10:05

jackscorrow

682
1
9
27

answered Jun 09 '11 at 03:59

anubhava

761,203
64
569
643

17

For Mac users: To prevent `extra characters at the end of d command` error you need to add semicolons before the closing brackets `sed '/^#/,/^\$/{/^#/!{/^\$/!d;};}' file.txt` – AvL Nov 12 '13 at 23:02
1

How would you do that if you want to include the # and $ lines for deletion? If you want to find $ at the end of a line, you can do $\$, – Timo Oct 30 '20 at 18:57
3

Then just use: `sed '/^#/,/^\$/d' file` – anubhava Oct 30 '20 at 19:34
I used `sed '/^====/,/^>>>>/d' file-with-git-merge-conflicts.xml > file-ok.xml` to remove git merge conflict lines between '=======' and '>>>>>>> branch-name, and `sed '/^<<< – bcag2 Apr 06 '22 at 07:40

score 62 · Answer 2 · answered Jan 24 '13 at 02:22

62

$ cat test
1
start
2
end
3
$ sed -n '1,/start/p;/end/,$p' test
1
start
end
3
$ sed '/start/,/end/d' test
1
3

answered Jan 24 '13 at 02:22

Lri

26,768
8
84
82

2

The speed at which this works on 300mb files is impressive. I'm talking split second on a SSD. – Ray Foss Apr 28 '15 at 19:15
2

I was a little confused since I'm not familiar with the sed syntax. It was not clear that the first and second sed commands did not have a dependency - i.e. the difference between the two is whether you want to preserve the match token or not. Until I tested it, I had assumed the first command removed everything between the tokens and the second removed the tokens themselves. If you're trying to strip out a chunk between tokens you only need to use the second command. – lukevp Feb 24 '19 at 21:56
No idea why, but `'1,/start/p;/end/,$p'` completely screwed up my workflow, as I was relying on this working. It does not work at all for me. – Akito Jan 31 '20 at 14:09
https://github.com/theAkito/akito-libbash/blob/dd91364083f13d1132d68489172bbce664b9c9c0/setup.bash#L37 is the line in question. Did I miss something? Because to me it looks like it is exactly as you have shown in your answer @Lri. – Akito Jan 31 '20 at 14:11
2

The solution that actually works is the following: `sed '/PATTERN-1/,/PATTERN-2/{//!d}' input.txt` – Akito Jan 31 '20 at 14:20
Thank you @Akito this works for me too in Fish shell – eyeseaevan May 21 '21 at 19:30

score 28 · Answer 3 · edited Sep 03 '18 at 23:45

In general form, if you have a file with contents of form abcde, where section a precedes pattern b, then section c precedes pattern d, then section e follows, and you apply the following sed commands, you get the following results.

In this demonstration, the output is represented by => abcde, where the letters show which sections would be in the output. Thus, ae shows an output of only sections a and e, ace would be sections a, c, and e, etc.

Note that if b or d appear in the output, those are the patterns appearing (i.e., they're treated as if they're sections in the output).

Also don't confuse the /d/ pattern with the command d. The command is always at the end in these demonstrations. The pattern is always between the //.

sed -n -e '/b/,/d/!p' abcde => ae
sed -n -e '/b/,/d/p' abcde => bcd
sed -n -e '/b/,/d/{//!p}' abcde => c
sed -n -e '/b/,/d/{//p}' abcde => bd
sed -e '/b/,/d/!d' abcde => bcd
sed -e '/b/,/d/d' abcde => ae
sed -e '/b/,/d/{//!d}' abcde => abde
sed -e '/b/,/d/{//d}' abcde => ace

Is there any way to get `ade`? – Tripp Kinetics May 31 '23 at 16:26 — Tripp Kinetics, May 31 '23 at 16:26

SLePort · Answer 4 · 2017-06-30T05:04:50.777

21

Another approach with sed:

sed '/^#/,/^\$/{//!d;};' file

/^#/,/^\$/: from line starting with # up to next line starting with $
//!d: delete all lines except those matching the address patterns

edited Jun 30 '17 at 05:04

answered Jun 29 '17 at 19:21

SLePort

15,211
3
34
44

1

how do you do this inclusive of the patterns? – qodeninja Dec 30 '17 at 01:53
8

Try this: `sed '/^#/,/^\$/d;' file`. – SLePort Dec 30 '17 at 05:28
How to delete including only the first or only the last pattern? – t7e Jun 03 '22 at 15:35

score 6 · Answer 5 · answered Jun 09 '11 at 03:34

I did something like this long time ago and it was something like:

sed -n -e "1,/# ID 1/ p" -e "/\$ description 1/,$ p"

Which is something like:

-n suppress all output
-e "1,/# ID 1/ p" execute from the first line until your pattern and p (print)
-e "/\$ description 1/,$ p" execute from the second pattern until the end and p (print).

I might be wrong with some of the escaping on the strings, so please double check.

score -1 · Answer 6 · answered Jan 30 '20 at 17:09

The example below removes lines between "if" and "end if".

All files are scanned, and lines between the two matching patterns are removed ( including them ).

IFS='
'
PATTERN_1="^if"
PATTERN_2="end if"

# Search for the 1st pattern in all files under the current directory.
GREP_RESULTS=(`grep -nRi "$PATTERN_1" .`)

# Go through each result
for line in "${GREP_RESULTS[@]}"; do

   # Save the file and line number where the match was found.
   FILE=${line%%:*}
   START_LINE=`echo "$line" | cut -f2 -d:`

   # Search on the same file for a match of the 2nd pattern. The search 
   # starts from the line where the 1st pattern was matched.
   GREP_RESULT=(`tail -n +${START_LINE} $FILE | grep -in "$PATTERN_2" | head -n1`)
   END_LINE="$(( $START_LINE + `echo "$GREP_RESULT" | cut -f1 -d:` - 1 ))"

   # Remove lines between first and second match from file
   sed -e "${START_LINE},${END_LINE}d;" $FILE > $FILE

done

I would not recommend your approach. While it demonstrates a couple of cool things like how to parse the output of grep in a clever way, your script runs several external programs and calls the shell functions several times when it can all be done internallly by sed, much more efficiently. Thus the downvote. — Rich, Jul 12 '22 at 22:08

Using sed to delete all lines between two matching patterns

6 Answers6

OUTPUT

Explanation:

Linked