Cross compare ArrayList elements and remove duplicates

Question

I have an ArrayList<MyObject> that may (or may not) contain duplicates of MyObject I need to remove from the List. How can I do this in a way that I don't have to check duplication twice as I would do if I were to iterate the list in two for-loops and cross checking every item with every other item.

I just need to check every item once, so comparing A:B is enough - I don't want to compare B:A again, as I already did that.

Furthermore; can I just remove duplicates from the list while looping? Or will that somehow break the list and my loop?

Edit: Okay, I forgot an important part looking through the first answers: A duplicate of MyObject is not just meant in the Java way meaning Object.equals(Object), but I need to be able to compare objects using my own algorithm, as the equality of MyObjects is calculated using an algorithm that checks the Object's fields in a special way that I need to implement!

Furthermore, I can't just override euqals in MyObject as there are several, different Algorithms that implement different strategies for checking the equality of two MyObjects - e.g. there is a simple HashComparer and a more complex EuclidDistanceComparer, both being AbstractComparers implementing different algorithms for the public abstract boolean isEqual(MyObject obj1, MyObject obj2);

Answer 1

Sort the list, and the duplicates will be adjacent to each other, making them easy to identify and remove. Just go through the list remembering the value of the previous item so you can compare it with the current one. If they are the same, remove the current item.

And if you use an ordinary for-loop to go through the list, you control the current position. That means that when you remove an item, you can decrement the position (n--) so that the next time around the loop will visit the same position (which will now be the next item).

You need to provide a custom comparison in your sort? That's not so hard:

Collections.sort(myArrayList, new Comparator<MyObject>() {

    public int compare(MyObject o1, MyObject o2) {
        return o1.getThing().compareTo(o2.getThing());
    }
});

I've written this example so that getThing().compareTo() stands in for whatever you want to do to compare the two objects. You must return an integer that is zero if they are the same, greater than 1 if o1 is greater than o2 and -1 if o1 is less than o2. If getThing() returned a String or a Date, you'd be all set because those classes have a compareTo method already. But you can put whatever code you need to in your custom Comparator.

Answer 2

Create a set and it will remove the duplicates automatically for you if the ordering is not important.

Set<MyObject> mySet = new HashSet<MyObject>(yourList);

Answer 3

Instantiate a new set-based collection HashSet. Don't forget to implement equals and hashcode for MyObject.

Good Luck!

Answer 4

If object order is insignificant

If the order is not important, you can put the elements of the list into a Set:

Set<MyObject> mySet = new HashSet<MyObject>(yourList);

The duplicates will be removed automatically.

If object order is significant

If ordering is significant, then you can manually check for duplicates, e.g. using this snippet:

// Copy the list.
ArrayList<String> newList = (ArrayList<String>) list.clone();

// Iterate
for (int i = 0; i < list.size(); i++) {
    for (int j = list.size() - 1; j >= i; j--) {
        // If i is j, then it's the same object and don't need to be compared.
        if (i == j) {
            continue;
        }
        // If the compared objects are equal, remove them from the copy and break
        // to the next loop
        if (list.get(i).equals(list.get(j))) {
            newList.remove(list.get(i));
            break;
        }
        System.out.println("" + i + "," + j + ": " + list.get(i) + "-" + list.get(j));
    }
}

This will remove all duplicates, leaving the last duplicate value as original entry. In addition, it will check each combination only once.

Using Java 8

Java Streams makes it even more elegant:

List<Integer> newList = oldList.stream()
    .distinct()
    .collect(Collectors.toList());

If you need to consider two of your objects equal based on your own definition, you could do the following:

public static <T, U> Predicate<T> distinctByProperty(Function<? super T, ?> propertyExtractor) {
    Set<Object> seen = ConcurrentHashMap.newKeySet();
    return t -> seen.add(propertyExtractor.apply(t));
}

(by Stuart Marks)

And then you could do this:

List<MyObject> newList = oldList.stream()
    .filter(distinctByProperty(t -> {
        // Your custom property to use when determining whether two objects
        // are equal. For example, consider two object equal if their name
        // starts with the same character.
        return t.getName().charAt(0);
    }))
    .collect(Collectors.toList());

Futhermore

You cannot modify a list while an Iterator (which is usually used in a for-each loop) is looping through an array. This will throw a ConcurrentModificationException. You can modify the array if you are looping it using a for loop. Then you must control the iterator position (decrementing it while removing an entry).

Answer 5

Or http://docs.oracle.com/javase/6/docs/api/java/util/SortedSet.html if you need sort-order..

EDIT: What about deriving from http://docs.oracle.com/javase/6/docs/api/java/util/TreeSet.html, it will allow you to pass in a Comparator at construction time. You override add() to use your Comparator instead of equals() - this will give you the flexibility of creating different sets that are ordered according to your Comparator and they will implement your "Equality"-Strategy.

Dont forget about equals() and hashCode() though...