I'd like to alphabetically lines between 2 patterns in a Bash shell script.
Given the following input file:
aaa
bbb
PATTERN1
foo
bar
baz
qux
PATTERN2
ccc
ddd
I expect as output:
aaa
bbb
PATTERN1
bar
baz
foo
qux
PATTERN2
ccc
ddd
Preferred tool is an AWK "one-liner". Sed and other solutions also accepted. It would be nice if an explanation is included.
This is a perfect case to use asort() to sort an array in GNU awk:
gawk '/PATTERN1/ {f=1; delete a}
/PATTERN2/ {f=0; n=asort(a); for (i=1;i<=n;i++) print a[i]}
!f
f{a[$0]=$0}' file
This uses a similar logic as How to select lines between two marker patterns which may occur multiple times with awk/sed with the addition that it:
Detailed explanation:
/PATTERN1/ {f=1; delete a} when finding a line matching PATTERN1, sets a flag on, and clears the array of lines./PATTERN2/ {f=0; n=asort(a); for (i=1;i<=n;i++) print a[i]} when finding a line matching PATTERN2, sets the flag off. Also, sorts the array a[] containing all the lines in the range and print them.!f if the flag is off (that is, outside the range), evaluate as True so that the line is printed.f{a[$0]=$0} if the flag is on, store the line in the array a[] so that its info can be used later on.▶ gawk '/PATTERN1/ {f=1} /PATTERN2/ {f=0; n=asort(a); for (i=1;i<=n;i++) print a[i]} !f; f{a[$0]=$0}' FILE
aaa
bbb
PATTERN1
bar
baz
foo
qux
PATTERN2
ccc
ddd
You can use sed with head and tail:
{
sed '1,/^PATTERN1$/!d' FILE
sed '/^PATTERN1$/,/^PATTERN2$/!d' FILE | head -n-1 | tail -n+2 | sort
sed '/^PATTERN2$/,$!d' FILE
} > output
The first line prints everything from the 1st line to PATTERN1.
The second line takes the lines between PATTERN1 and PATTERN2, removes the last and first line, and sorts the remaining lines.
The third line prints everything from PATTERN2 to the end of the file.
More complicated, but may ease the memory load of storing lots of lines (your cfg file would have to be pretty huge for this to matter, but nevertheless...). Using GNU awk and a sort coprocess:
gawk -v p=1 '
/^PATTERN2/ { # when we we see the 2nd marker:
# close the "write" end of the pipe to sort. Then sort will know it
# has all the data and it can begin sorting
close("sort", "to");
# then sort will print out the sorted results, so read and print that
while (("sort" |& getline line) >0) print line
# and turn the boolean back to true
p=1
}
p {print} # if p is true, print the line
!p {print |& "sort"} # if p is false, send the line to `sort`
/^PATTERN1/ {p=0} # when we see the first marker, turn off printing
' FILE
It's a little unconventional but using Vim:
vim -c 'exe "normal /PATTERN1\<cr>jV/PATTERN2\<cr>k: ! sort\<cr>" | wq!' FILE
Where \<cr> is a carriage return, entered as CTRL-v then CTRL-M.
Further explanation:
/PATTERN1\<cr> - search for the first patternj - go to the next lineV - enter visual mode/PATTERN2\<cr> - search for the second patternk - go back one line: ! sort\<cr> - sort the visual text you just selectedwq! - save and exit
Obviously this is inferior to the GNU AWK solution, but all the same, this is a GNU sed solution:
sed '
/PATTERN1/,/PATTERN2/ {
/PATTERN1/b # branch/break if /PATTERN1/. This line is printed
/PATTERN2/ { # if /PATTERN2/,
x # swap hold and pattern spaces
s/^\n// # delete the leading newline. The first H puts it there
s/.*/sort <<< "&"/e # sort the pattern space by calling Unix sort
p # print the sorted pattern space
x # swap hold and pattern space again to retrieve PATTERN2
p # print it also
}
H # Append the pattern space to the hold space
d # delete this line for now - it will be printed in the block above
}
' FILE
Note that I rely on the e command, a GNU extension.
Testing:
▶ gsed '
/PATTERN1/,/PATTERN2/ {
/PATTERN1/b
/PATTERN2/ {
x
s/^\n//; s/.*/sort <<< "&"/ep
x
p
}
H
d
}
' FILE
aaa
bbb
PATTERN1
bar
baz
foo
qux
PATTERN2
ccc
ddd
Here is a small and easy to understand shell script for sorting lines between two patterns:
#!/bin/sh
in_file=$1
out_file=$2
temp_file_for_sort="$out_file.temp.for_sort"
curr_state=0
in_between_count=0
rm -rf $out_file
while IFS='' read -r line; do
if (( $curr_state == 0 )); then
#write this line to output
echo $line >> $out_file
is_start_line=`echo $line | grep "^PATTERN_START$"`
if [ -z "$is_start_line" ]; then
continue
else
rm -rf $temp_file_for_sort
in_between_count=0
curr_state=1
fi
else
is_end_line=`echo $line | grep "^PATTERN_END"`
if [ -z "$is_end_line" ]; then
#Line inside block - to be sorted
echo $line >> $temp_file_for_sort
in_between_count=$(( $in_between_count +1 ))
else
#End of block
curr_state=0
if (( $in_between_count != 0 )); then
sort -o $temp_file_for_sort $temp_file_for_sort
cat $temp_file_for_sort >> $out_file
rm -rf $temp_file_for_sort
fi
echo $line >> $out_file
fi
fi
done < $temp_file
#if something remains
if [ -f $temp_file_for_sort ]; then
cat $temp_file_for_sort >> $out_file
fi
rm -rf $temp_file_for_sort
Usage: <script_path> <input_file> <output_file>.
Pattern is hardcoded in file, can be changed as required (or taken as argument). Also, it creates a temporary file to sort intermediate data (<output_file>.temp.for_sort)
Algorithm:
Start with state = 0 and read the file line by line.
In state 0, line is written to output file and if START_PATTERN is encountered, state is set to 1.
In state 1, if line is not STOP_PATTERN, write line to temporary file In state 1, if line is STOP_PATTERN, sort temporary file, append temporary file contents to output file (and remove temporary file) and write STOP_PATTERN to output file. Also, change state to 0.
At last if something is left in temporary file (case when STOP_PATTERN is missing), write contents of temporary file to output file
Along the lines of the solution proposed by @choroba, using GNU sed (depends on Q command):
{
sed -n '1,/PATTERN1/p' FILE
sed '1,/PATTERN1/d; /PATTERN2/Q' FILE | sort
sed -n '/PATTERN2/,$p' FILE
}
Explanation:
p prints a line in the range 1 to /PATTERN1/ inclusive and ($ is end of file) in '1,/PATTERN1/p' and /PATTERN2/,$p respectively.-n disables default behaviour of printing all lines. Useful in conjunction with p.d command is used to delete lines 1 to the /PATTERN1/ and also to Q (quit without printing, GNU sed only) on the first line matching /PATTERN2/. These are the lines to be sorted, and are thus fed into sort.
This can also be done with non-GNU awk and system command sort, make it work on both macOS and Linux.
awk -v SP='PATTERN1' -v EP='PATTERN2' -v cmd=sort '{
if (match($0, SP)>0) {flag=1}
else if (match($0, EP)>0) {
for (j=0;j<length(a);j++) {print a[j]|cmd}
close(cmd); delete a; i=0; flag=0}
else if (flag==1) {a[i++]=$0; next}
print $0
}' FILE
Output:
aaa
bbb
PATTERN1
bar
baz
foo
qux
PATTERN2
ccc
ddd
