Remove list of items from another file in bash

Question

What would be the most efficient method (no repeated command execution) to remove items listed in one file from another file (unordered) ?

One can easily get the list of non-matching items in the second file by

cat first_file.txt second_file.txt | sort | uniq -u

but that would also contain all unmatched items from the first file too... now what?

score 4 · Accepted Answer · answered Sep 16 '15 at 14:37

This awk program takes a single pass through each file:

awk '
    NR == FNR {f1[$0] = 1; next}
    !($0 in f1)
' file1 file2

comm is useful for this job. It does require it's input files to be sorted:

# output lines unique to file2
comm -13 <(sort file1) <(sort file2)

1 Answers1