How to check if exists any duplicate in Java 8 Streams?

Question

In java 8, what's the best way to check if a List contains any duplicate?

My idea was something like:

list.size() != list.stream().distinct().count()

Is it the best way?

Answer 1

Your code would need to iterate over all elements. If you want to make sure that there are no duplicates simple method like

public static <T> boolean areAllUnique(List<T> list){
    Set<T> set = new HashSet<>();

    for (T t: list){
        if (!set.add(t))
            return false;
    }
    
    return true;
}

would be more efficient since it can give you false immediately when first non-unique element would be found.

This method could also be rewritten using Stream#allMatch which also is short-circuit (returns false immediately for first element which doesn't fulfill provided condition)
(assuming non-parallel streams and thread-safe environment)

public static <T> boolean areAllUnique(List<T> list){
    Set<T> set = new HashSet<>();
    return list.stream().allMatch(t -> set.add(t));
}

which can be farther shortened as @Holger pointed out in comment

public static <T> boolean areAllUnique(List<T> list){
    return list.stream().allMatch(new HashSet<>()::add);
}

Answer 2

I used the following:
1. return list.size() == new HashSet<>(list).size();.

I'm not sure how it compares to:
2. return list.size() == list.stream().distinct().count();
and
3. return list.stream().sequential().allMatch(new HashSet<>()::add);
in terms of performance.

The last one (#3) has possibility to handle not only collections (e.g. lists), but also streams (without explicitly collecting them).

Upd.: The last one (#3) seems to be the best not only because it can handle pure streams, but also because it stops on the first duplicate (while #1 and #2 always iterate till the end) — as @Pshemo said in comment.

Answer 3

You can use the counting collector.

Stream.of(1, 3, 4, 6, 7, 5, 6)
            .collect(Collectors.groupingBy(
                    Function.identity(), Collectors.counting()))
            .entrySet().stream().anyMatch(e -> e.getValue() > 1)

Answer 4

Given array arr,

arr.length != Arrays.stream(arr).distinct().count()

will help check for duplicates

Answer 5

Started this class as a StreamTool, but I think there must be an even better way with reduce or similar:

public class StreamTool {

    /**
     * Whether stream records are unique in that stream.
     * @param <T> Type of records
     * @param records
     * @return true if there are no duplicates, false otherwise
     */
    public static <T> boolean isUnique(Stream<T> records) {
        return records.allMatch(new HashSet<>()::add);
    }
}

Answer 6

Use set.add() it is faster.

Set<T> items = new HashSet<>();
list.stream().filter(n -> !items.add(n)) 
            .collect(Collectors.toSet());