Compare/Difference of two arrays in Bash

Question

Is it possible to take the difference of two arrays in Bash. What is a good way to do it?

Code:

Array1=( "key1" "key2" "key3" "key4" "key5" "key6" "key7" "key8" "key9" "key10" )
Array2=( "key1" "key2" "key3" "key4" "key5" "key6" ) 

Array3 =diff(Array1, Array2)

Array3 ideally should be :
Array3=( "key7" "key8" "key9" "key10" )

Having skimmed over the solutions, I decided not to use arrays in cases where I've got to diff them. — x-yuri, Oct 24 '18 at 18:07

score 202 · Answer 1 · answered Jan 27 '15 at 00:55

202

echo ${Array1[@]} ${Array2[@]} | tr ' ' '\n' | sort | uniq -u

Output

key10
key7
key8
key9

You can add sorting if you need

answered Jan 27 '15 at 00:55

Ilya Bystrov

2,902
2
14
21

58

He came in, he bossed it and he left. For anyone wondering how to save the value to an array, try this: `Array3=(\`echo ${Array1[@]} ${Array2[@]} | tr ' ' '\n' | sort | uniq -u \`)` – Anake Aug 22 '15 at 12:33
11

This is what shell programming is about. Keep it simple, use the tools available. If you want to implement the other solutions, you can, but you may have an easier time using a more robust language. – wrangler Aug 26 '15 at 14:10
Incredible even to this day. – Zee Feb 25 '16 at 20:29
25

Brilliant. Additional note for those who need the **asymmetrical** difference. You can get it by outputting the duplicates of the **symmetrical** difference and the Array you are interested in. IE if you want the values present in Array2, but not in Array1. `echo ${Array2[@]} ${Array3[@]} | tr ' ' '\n' | sort | uniq -D | uniq`, where Array3 is the output of the above. Additionally if you remove the array notations and assume the variables are space separated strings, this approach is posix shell compliant. – Arwyn Mar 29 '16 at 06:12
9

Awesome solution. Slight improvement if array elements might contain spaces: `printf '%s\n' "${Array1[@]}" "${Array2[@]}" | sort | uniq -u` – misberner Jul 13 '16 at 17:00
23

To simplify @Arwyn's suggestion, you can add the ignored array twice to ensure only the differences in Array2 are shown. `echo ${Array1[@]} ${Array1[@]} ${Array2[@]} | tr ' ' '\n' | sort | uniq -u` – Christopher Markieta Aug 06 '17 at 19:36
6

one small comment to @ChristopherMarkieta's answer: the question was to calculate Array1-Array2, in this case it should be `echo ${Array1[@]} ${Array2[@]} ${Array2[@]} | tr ' ' '\n' | sort | uniq -u` (Array2 two times). And thanks for great addition to a great answer. – astafev.evgeny Dec 14 '17 at 08:14
2

Adding to the expansion of @misberner: If your array contains newline and white spaces: `printf "%s\0" "${Array1[@]}" "${Array2[@]}" | sort -z | uniq -zu` . (The output will be null-delimted and needs to be processed accordingly) – kvantour May 11 '20 at 12:00
I have not been able to get any of this thread's suggestions to work in my function. Lend a hand if you are able? https://stackoverflow.com/questions/63314441/diff-two-arrays-each-containing-files-paths-into-a-third-array-for-removal – JamesIsIn Aug 08 '20 at 10:40
Found an edge case where this solution does not work - if both arrays are empty, doing ```echo ${Array1[@]} ${Array2[@]} | tr ' ' '\n' | sort | uniq -u | wc -l ``` prints 1, not 0 – kp123 Sep 25 '20 at 05:42
2

Actually, the answer as given is computing the symmetric difference. Things that are in both sets will be removed, things only in one will be kept: ```$ echo {1..4} {2..6} | tr ' ' '\n' | sort | uniq -u | tr '\n' ' '``` gives ```1 5 6```. Including the second set twice makes a regular difference: ```$ echo {1..4} {2..6} {2..6} | tr ' ' '\n' | sort | uniq -u | tr '\n' ' '``` gives ```1```. – jspencer Jun 03 '21 at 22:33
@ilya-bystrov you should expand your anwser with the jspencer explanation – Ozeta Oct 07 '21 at 22:37

score 47 · Accepted Answer · edited Sep 09 '18 at 12:49

47

If you strictly want Array1 - Array2, then

Array1=( "key1" "key2" "key3" "key4" "key5" "key6" "key7" "key8" "key9" "key10" )
Array2=( "key1" "key2" "key3" "key4" "key5" "key6" )

Array3=()
for i in "${Array1[@]}"; do
    skip=
    for j in "${Array2[@]}"; do
        [[ $i == $j ]] && { skip=1; break; }
    done
    [[ -n $skip ]] || Array3+=("$i")
done
declare -p Array3

Runtime might be improved with associative arrays, but I personally wouldn't bother. If you're manipulating enough data for that to matter, shell is the wrong tool.

For a symmetric difference like Dennis's answer, existing tools like comm work, as long as we massage the input and output a bit (since they work on line-based files, not shell variables).

Here, we tell the shell to use newlines to join the array into a single string, and discard tabs when reading lines from comm back into an array.

$ oldIFS=$IFS IFS=$'\n\t'
$ Array3=($(comm -3 <(echo "${Array1[*]}") <(echo "${Array2[*]}")))
comm: file 1 is not in sorted order
$ IFS=$oldIFS
$ declare -p Array3
declare -a Array3='([0]="key7" [1]="key8" [2]="key9" [3]="key10")'

It complains because, by lexographical sorting, key1 < … < key9 > key10. But since both input arrays are sorted similarly, it's fine to ignore that warning. You can use --nocheck-order to get rid of the warning, or add a | sort -u inside the <(…) process substitution if you can't guarantee order&uniqueness of the input arrays.

edited Sep 09 '18 at 12:49

030

10,842
12
78
123

answered Feb 23 '10 at 01:00

ephemient

198,619
38
280
391

1

+1 for the 1st snippet, which also works with elements with embedded whitespace. The 2nd snippet works with elements with embedded _spaces_ only. You can do away with saving and restoring `$IFS` if you simply prepend `IFS=$'\n\t' ` directly to the `Array3=...` command. – mklement0 Jun 01 '14 at 05:35
2

@mklement0 The command you're suggesting: `IFS=$'\n\t' Array3=( ... )` _will_ set `IFS` globally. Try it! – gniourf_gniourf Jun 01 '14 at 07:06
@gniourf_gniourf: Thanks for catching that! Because my fallacy may be seductive to others too, I'll leave my original comment and explain here: While it's a common and useful idiom to prepend an _ad-hoc, command-local variable assignment_ to a simple command, it does NOT work here, because my _command is composed entirely of assignments_. _No command name_ (external executable, builtin) follows the assignments, which makes _all_ of them _global_ (in the context of the current shell); see `man bash`, section `SIMPLE COMMAND EXPANSION`). – mklement0 Jun 01 '14 at 16:25
Can you give an example how to do this in a C-shell (csh)? – Stefan Nov 11 '14 at 15:13
@Stefan: Ugh, csh should never be used. `set Array3 = ( )` `foreach i ( $Array1 )` `set skip = 0` `foreach j ( $Array2 )` `if ( "$i" == "$j" ) then` `set skip = 1` `break` `endif` `end` `if ( "$skip" == 0 ) then` `set Array3 = ( $Array3:q "$i" )` `endif` `end` All the control statements need to be on their own lines. – ephemient Nov 11 '14 at 22:00
would this work if array 1 had an extra value thats not inside array2? for me if its not in array2 then its not needed anyway, so i need it to ignore that case. its breaking at the moment and im wondering if thats why (edit: seems like it does actually, thank you) – blamb Jul 11 '23 at 19:20

SiegeX · Answer 3 · 2010-02-24T22:26:41.427

15

Anytime a question pops up dealing with unique values that may not be sorted, my mind immediately goes to awk. Here is my take on it.

Code

#!/bin/bash

diff(){
  awk 'BEGIN{RS=ORS=" "}
       {NR==FNR?a[$0]++:a[$0]--}
       END{for(k in a)if(a[k])print k}' <(echo -n "${!1}") <(echo -n "${!2}")
}

Array1=( "key1" "key2" "key3" "key4" "key5" "key6" "key7" "key8" "key9" "key10" )
Array2=( "key1" "key2" "key3" "key4" "key5" "key6" )
Array3=($(diff Array1[@] Array2[@]))
echo ${Array3[@]}

Output

$ ./diffArray.sh
key10 key7 key8 key9

*Note**: Like other answers given, if there are duplicate keys in an array they will only be reported once; this may or may not be the behavior you are looking for. The awk code to handle that is messier and not as clean.

edited Feb 24 '10 at 22:26

answered Feb 24 '10 at 22:10

SiegeX

135,741
24
144
154

To summarize the behavior and constraints: (a) performs a _symmetrical_ difference: outputs a _single_ array with elements unique to _either_ input array (which with the OP's sample data happens to be the same as only outputting elements unique to the _first_ array), (b) only works with elements that have no embedded whitespace (which satisfies the OP's requirements), and (c) the order of elements in the output array has NO guaranteed relationship to the order of input elements, due to `awk`'s unconditional use of _associative_ arrays - as evidenced by the sample output. – mklement0 Jun 01 '14 at 05:46
Also, this answer uses a clever-and-noteworthy-but-baffling-if-unexplained workaround for bash's lack of support for passing _arrays_ as arguments: `Array1[@]` and `Array2[@]` are passed as _strings_ - the respective array names plus the all-subscripts suffix `[@]`- to shell function `diff()` (as arguments `$1` and `$2`, as usual). The shell function then uses bash's variable _indirection_ (`{!...}`) to _indirectly_ refer to all elements of the original arrays (`${!1}` and `${!1}'). – mklement0 Jun 01 '14 at 05:48
how to transform a string "a b C" into an array? – brauliobo Oct 07 '14 at 21:46
found an error: elements in `Array2` not in `Array1` will show in `diff()` – brauliobo Oct 07 '14 at 21:58
This solution doesn't work for array elements containing whitespace. The example script can fail in multiple ways due to unquoted strings being GLOB expanded by the shell. It fails if you do `touch Array1@` before you run the script, because the strings `Array1[@]` and `Array2[@]` are used as unquoted shell GLOB patterns. It fails if one array contains the element `*` because that unquoted GLOB pattern matches all the files in the current directory. – Ian D. Allen Mar 22 '21 at 17:06

score 11 · Answer 4 · answered Feb 22 '17 at 18:28

Having ARR1 and ARR2 as arguments, use comm to do the job and mapfile to put it back into RESULT array:

ARR1=("key1" "key2" "key3" "key4" "key5" "key6" "key7" "key8" "key9" "key10")
ARR2=("key1" "key2" "key3" "key4" "key5" "key6")

mapfile -t RESULT < \
    <(comm -23 \
        <(IFS=$'\n'; echo "${ARR1[*]}" | sort) \
        <(IFS=$'\n'; echo "${ARR2[*]}" | sort) \
    )

echo "${RESULT[@]}" # outputs "key10 key7 key8 key9"

Note that result may not meet source order.

Bonus aka "that's what you are here for":

function array_diff {
    eval local ARR1=\(\"\${$2[@]}\"\)
    eval local ARR2=\(\"\${$3[@]}\"\)
    local IFS=$'\n'
    mapfile -t $1 < <(comm -23 <(echo "${ARR1[*]}" | sort) <(echo "${ARR2[*]}" | sort))
}

# usage:
array_diff RESULT ARR1 ARR2
echo "${RESULT[@]}" # outputs "key10 key7 key8 key9"

Using those tricky evals is the least worst option among others dealing with array parameters passing in bash.

Also, take a look at comm manpage; based on this code it's very easy to implement, for example, array_intersect: just use -12 as comm options.

@lantrix, `mapfile` can be easily replaced with `while..read`, and even totally cut if one doesn't need an array as a result. All the magic happens in `comm`. — Alex Offshore, Jul 11 '17 at 09:48

score 9 · Answer 5 · edited Jun 02 '14 at 03:10

9

In Bash 4:

declare -A temp    # associative array
for element in "${Array1[@]}" "${Array2[@]}"
do
    ((temp[$element]++))
done
for element in "${!temp[@]}"
do
    if (( ${temp[$element]} > 1 ))
    then
        unset "temp[$element]"
    fi
done
Array3=(${!temp[@]})    # retrieve the keys as values

Edit:

ephemient pointed out a potentially serious bug. If an element exists in one array with one or more duplicates and doesn't exist at all in the other array, it will be incorrectly removed from the list of unique values. The version below attempts to handle that situation.

declare -A temp1 temp2    # associative arrays
for element in "${Array1[@]}"
do
    ((temp1[$element]++))
done

for element in "${Array2[@]}"
do
    ((temp2[$element]++))
done

for element in "${!temp1[@]}"
do
    if (( ${temp1[$element]} >= 1 && ${temp2[$element]-0} >= 1 ))
    then
        unset "temp1[$element]" "temp2[$element]"
    fi
done
Array3=(${!temp1[@]} ${!temp2[@]})

edited Jun 02 '14 at 03:10

mklement0

382,024
64
607
775

answered Feb 22 '10 at 18:48

Dennis Williamson

346,391
90
374
439

That performs a symmetric difference, and assumes that the original arrays have no duplicates. So it's not what I would have thought of first, but it works well for OP's one example. – ephemient Feb 22 '10 at 19:26
@ephemient: Right, the parallel would be to `diff(1)` which is also symmetric. Also, this script will work to find elements unique to any number of arrays simply by adding them to the list in the second line of the first version. I've added an edit which provides a version to handle duplicates in one array which don't appear in the other. – Dennis Williamson Feb 22 '10 at 21:02
Thanks A lot.. I was thinking if there was any obvious way of doing it.. If i am not aware of any command which would readily give the diff of 2 arrays.. Thanks for your support and help. I modified the code to read the diff of 2 files which was little easier to program – Kiran Feb 22 '10 at 23:21
Your 2nd snippet won't work, because `>` only works in `(( ... ))`, not in `[[ ... ]]`; in the latter, it'd have to be `-gt`; however, since you probably meant `>=` rather than `>`, `>` should be replaced with `-ge`. To be explicit about what "symmetric" means in this context: the output is a _single_ array containing values that are unique to _either_ array. – mklement0 Jun 01 '14 at 05:16
1

@mklement0: `>` does work inside double square brackets, but lexically rather than numerically. Because of that, when comparing integers, double parentheses should be used - so you are correct in that regard. I've updated my answer accordingly. – Dennis Williamson Jun 01 '14 at 14:18
@DennisWilliamson: Thanks for the clarification re `>` inside `[[ ... ]]` and thanks for updating your answer. However, I think it should be `>= 1` rather than `> 1`. More crucially, though, the `((...))` conditional will break with non-existent (empty) `temp2` elements, so you either need to use `${temp2[$element]-0}` or stick with `[[...]]` and `-ge`. – mklement0 Jun 01 '14 at 14:51
Since I didn't hear back and didn't want a demonstrably broken answer to stand, I've taken the liberty to fix it. Please let me know if you feel the fix is incorrect or inappropriate. On a general note, this answer only works with array elements without embedded whitespace (which does satisfy the OP's requirements as stated). – mklement0 Jun 02 '14 at 03:10

score 8 · Answer 6 · edited May 23 '17 at 11:47

8

It is possible to use regex too (based on another answer: Array intersection in bash):

list1=( 1 2 3 4   6 7 8 9 10 11 12)
list2=( 1 2 3   5 6   8 9    11 )

l2=" ${list2[*]} "                    # add framing blanks
for item in ${list1[@]}; do
  if ! [[ $l2 =~ " $item " ]] ; then    # use $item as regexp
    result+=($item)
  fi
done
echo  ${result[@]}:

Result:

$ bash diff-arrays.sh 
4 7 10 12

edited May 23 '17 at 11:47

Community

1
1

answered Feb 15 '16 at 13:06

Denis Gois

91
1
3

2

seems odd that this was down voted with no comment. If there's a problem with it, do everyone a favor and point out what the problem is. – philwalk Mar 09 '17 at 17:57

score 3 · Answer 7 · answered Feb 23 '10 at 01:24

Array1=( "key1" "key2" "key3" "key4" "key5" "key6" "key7" "key8" "key9" "key10" )
Array2=( "key1" "key2" "key3" "key4" "key5" "key6" )
Array3=( "key1" "key2" "key3" "key4" "key5" "key6" "key11" )
a1=${Array1[@]};a2=${Array2[@]}; a3=${Array3[@]}
diff(){
    a1="$1"
    a2="$2"
    awk -va1="$a1" -va2="$a2" '
     BEGIN{
       m= split(a1, A1," ")
       n= split(a2, t," ")
       for(i=1;i<=n;i++) { A2[t[i]] }
       for (i=1;i<=m;i++){
            if( ! (A1[i] in A2)  ){
                printf A1[i]" "
            }
        }
    }'
}
Array4=( $(diff "$a1" "$a2") )  #compare a1 against a2
echo "Array4: ${Array4[@]}"
Array4=( $(diff "$a3" "$a1") )  #compare a3 against a1
echo "Array4: ${Array4[@]}"

output

$ ./shell.sh
Array4: key7 key8 key9 key10
Array4: key11

PhrozenByte · Answer 8 · 2023-04-14T14:51:19.730

@ilya-bystrov's most upvoted answer calculates the difference of Array1 and Array2. Please note that this is not the same as removing items from Array1 that are also in Array2. @ilya-bystrov's solution rather concatenates both lists and removes non-unique values. This is a huge difference when Array2 includes items that are not in Array1: Array3 will contain values that are in Array2, but not in Array1.

Here's a pure Bash solution for removing items from Array1 that are also in Array2 (note the additional "key11" in Array2):

Array1=( "key1" "key2" "key3" "key4" "key5" "key6" "key7" "key8" "key9" "key10" )
Array2=( "key1" "key2" "key3" "key4" "key5" "key6" "key11" )
Array3=( $(printf "%s\n" "${Array1[@]}" "${Array2[@]}" "${Array2[@]}" | sort | uniq -u) )

Array3 will consist of "key7" "key8" "key9" "key10" and exclude the unexpected "key11" when trying to remove items from Array1.

If your array items might contain whitespaces, use mapfile to construct Array3 instead, as suggested by @David:

Array1=( "key1" "key2" "key3" "key4" "key5" "key6" "key7" "key8" "key9" "key10" "key10" )
Array2=( "key1" "key2" "key3" "key4" "key5" "key6" "key11" )
mapfile -t Array3 < <(printf "%s\n" "${Array1[@]}" "${Array2[@]}" "${Array2[@]}" | sort | uniq -u)

Please note: This assumes that all values in Array1 are unique. Otherwise they won't show up in Array3. If Array1 contains duplicate values, you must remove the duplicates first (note the duplicate "key10" in Array1; possibly use mapfile if your items contain whitespaces):

Array1=( "key1" "key2" "key3" "key4" "key5" "key6" "key7" "key8" "key9" "key10" "key10" )
Array2=( "key1" "key2" "key3" "key4" "key5" "key6" "key11" )
Array3=( $({ printf "%s\n" "${Array1[@]} | sort -u; printf "%s\n" "${Array2[@]}" "${Array2[@]}"; } | sort | uniq -u) )

If you want to replicate the duplicates in Array1 to Array2, go with @ephemient' accepted answer. The same is true if Array1 and Array2 are huge: this is a very inefficient solution for a lot of items, even though it's negligible for a few items (<100). If you need to process huge arrays don't use Bash.

Nice! You should use `mapfile` to build `Array3` though, as the *sloppy* `A=(..)` expansion could break on whitespaces: `a=(string\ 1 string\ 2 string\ 3); b=(string\ 2);` `mapfile -t c < <(printf "%s\n" "${a[@]}" "${b[@]}" "${b[@]}" | sort | uniq -u);` — David, Dec 14 '22 at 09:24

Netwons · Answer 9 · 2022-01-08T20:18:31.277

0

this code replace with diff

echo ${test1[@]} ${test2[@]} | sed 's/ /\n/g' | sort | uniq -u

for result reverse use uniq -d

edited Jan 08 '22 at 20:18

answered Jan 08 '22 at 20:06

Netwons

1,170
11
14

Compare/Difference of two arrays in Bash

9 Answers9

Code

Output

Linked

Related