Remove duplicates from list based on some property

Question

I need to remove duplicate objects from a list based on some properties and have distinct values.

List<Order> orderList = new ArrayList<>();

I need to remove the objects having same id and value and having status 'COMPLETE'. I need distinct values of this combination: id, value and status = COMPLETE My code is like this:

private static <T> Predicate<T> distinctByKeys(Function<? super T, ?>... keyExtractors) {
    final Map<List<?>, Boolean> seen = new ConcurrentHashMap<>();
 
return t -> 
{
  final List<?> keys = Arrays.stream(keyExtractors)
              .map(ke -> ke.apply(t))
              .collect(Collectors.toList());
   
  return seen.putIfAbsent(keys, Boolean.TRUE) == null;
};
}

But I need to filter based on three conditions: same id, value and status = 'COMPLETE'. Example:

Order object1: id=1, value="test", status =COMPLETE
Order object2: id=2, value="abc", status =INCOMPLETE
Order object3: id=1, value="test", status =COMPLETE
Order object4: id=1, value="test", status =COMPLETE

Output:
Order object1: id=1, value="test", status =COMPLETE
Order object2: id=2, value="abc", status =INCOMPLETE

Any ideas on this?

I need to filter based on three conditions: same id, value and status = 'COMPLETE' — Krishnanunni P V, Apr 13 '21 at 07:03
Use a map as intermediate store and your property combinations as the key. — Thomas, Apr 13 '21 at 07:05
@Stultuske I'm able to find distinct values using above method. But how to add status =Complete also in that filter? — Krishnanunni P V, Apr 13 '21 at 07:07
@Thomas the above method does the same. But how to include status = COMPLETE in that? — Krishnanunni P V, Apr 13 '21 at 07:09
you are aware you can have several conditions in a single conditional statement? — Stultuske, Apr 13 '21 at 07:10
Well if only status COMPLETE should be equal just check for that and map anything else to a synthetic "unique" element. — Thomas, Apr 13 '21 at 07:11
@Thomas I need distinct values of this combination: id, value and status = COMPLETE — Krishnanunni P V, Apr 13 '21 at 07:13
And how do you want to handle those with status != Complete? — Thomas, Apr 13 '21 at 07:14
Then first use `.filter()` to only keep those with status == COMPLETE. — Thomas, Apr 13 '21 at 07:19
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/231054/discussion-between-krishnanunni-p-v-and-thomas). — Krishnanunni P V, Apr 13 '21 at 07:24
Well, you should be clear then. So `status != COMPLETE` should be unique in any case even with the same id and value? That's where "map to a synthetic unique status" comes in (you could use `Object` and either store `status` if it is complete or `new Object()` in other cases). — Thomas, Apr 13 '21 at 07:24

the Hutt · Answer 1 · 2021-04-13T10:30:14.830

To remove items with COMPLETE status use filter method. To remove duplicates use distinct method.
In order to use distinct your object needs to implement equals method. Following is the demo code:

public class Test {
    public static void main(String[] args) throws Exception {
        List<DataObject> list = List.of(new DataObject(1,"abc","processing"),
                        new DataObject(2,"bcd","complete"),
                        new DataObject(1,"abc","processing"),
                        new DataObject(3,"abf","processing"));
        
        List<DataObject> result = list.stream()
                        .filter(d-> !d.getStatus().equals("complete"))
                        .distinct()
                        .collect(Collectors.toList());
        
        System.out.println(result);
    }
    
}

class DataObject {
    private int id;
    private String data;
    private String status;
    
    public DataObject(int id, String data, String status) {
        this.id = id;
        this.data = data;
        this.status = status;
    }
    
    @Override
    public int hashCode() {
        final int prime = 31;
        int result = 1;
        result = prime * result + ((data == null) ? 0 : data.hashCode());
        result = prime * result + id;
        return result;
    }

    //this is eclipse autogenerated code
   //modify conditions as you want
    @Override
    public boolean equals(Object obj) {
        if (this == obj)
            return true;
        if (obj == null)
            return false;
        if (getClass() != obj.getClass())
            return false;
        DataObject other = (DataObject) obj;
        if (data == null) {
            if (other.data != null)
                return false;
        } else if (!data.equals(other.data))
            return false;
        if (id != other.id)
            return false;
        return true;
    }
    
    @Override
    public String toString() {
        return "id: " + id;
    }

    public int getId() {
        return id;
    }
    
    public String getData() {
        return data;
    }
    
    public String getStatus() {
        return status;
    }
    
}

Output:

[id: 1, id: 3]

If you don't want to implement equals then you can use this method after filter operation.

Edit: updating the solution as question has changed.
Create a wrapper object which will have equals method based on your needs:

public class Test {
    public static void main(String[] args) throws Exception {
        List<DataObject> list = List.of(new DataObject(1,"abc","complete"),
                        new DataObject(2,"bcd","complete"),
                        new DataObject(1,"abc","complete"),
                        new DataObject(3,"aaa","complete"));
        
        List<DataObject> result = list.stream()
                        .map(Wrapper::new)
                        .distinct()
                        .map(Wrapper::getContent)
                        .collect(Collectors.toList());
        
        System.out.println(result);
    }
    
}

class Wrapper {
    DataObject dto;
    public Wrapper(DataObject dto) {
        this.dto = dto;
    }
    
    public DataObject getContent() {
        return this.dto;
    }
    
    @Override
    public int hashCode() {
        //return this.dto.hashCode();
        //considering id is unique and you haven't implemented hashCode
        return this.dto.getId();
    }
    
    @Override
    public boolean equals(Object obj) {
        //Todo: improve the implementation as per your needs
        
        Wrapper w = (Wrapper) obj;
        DataObject other = w.getContent(); 
        
        if (!this.dto.getData().equals(other.getData()))
            return false;
        if (this.dto.getId() != other.getId())
            return false;
        if (this.dto.getStatus() != other.getStatus())
            return false;
        return true;
    }
}

class DataObject {
    private int id;
    private String data;
    private String status;
    
    public DataObject(int id, String data, String status) {
        this.id = id;
        this.data = data;
        this.status = status;
    }
    
    @Override
    public String toString() {
        return "id: " + id;
    }

    public int getId() {
        return id;
    }
    
    public String getData() {
        return data;
    }
    
    public String getStatus() {
        return status;
    }
    
}

Output:

[id: 1, id: 2, id: 3]

I don't want to filter out complete status. Please see my edit in question — Krishnanunni P V, Apr 13 '21 at 09:54
Then from my answer, remove `filter` clause and in class's `equals` method compare `status` as well. Alternatively you can create a wrapper object which will check this, I'll update the answer shortly. — the Hutt, Apr 13 '21 at 10:07

Nikolas Charalambidis · Answer 2 · 2021-04-16T06:10:03.240

There is unfortunately no built-in method to distinct a collection based on criteria. You need to group the list in two nested maps to find out the duplicated objects. It's structure can be one of these:

Map<Integer, Map<String, List<Order>> where Integer is the id and String is the value.
Map<String, Map<Integer, List<Order>> where Integer is the id and String is the value.

Once grouped, you easily detect the same objects by your criteria because each list will contain two or more of such objects. All you need is filter and flatMap to get the duplicated objects into List<Order>.

Grouping

Map<Integer, Map<String, List<Order>>> maps = orderList.stream()
         .filter(order -> order.getState() == State.COMPLETED)
         .collect(Collectors.groupingBy(
                 Order::getId,
                 Collectors.groupingBy(Order::getValue)));

Extraction

List<Order> duplicates = maps.entrySet().stream()
        .flatMap(map -> map.getValue().entrySet().stream()
            .filter(entry -> entry.getValue().size() > 1)
            .flatMap(entry -> entry.getValue().stream()))
        .collect(Collectors.toList());

Remove the duplicates from the original list.
```
orderList.removeAll(duplicates);
```

Edit: I have noticed the question was edited and the problem is slightly different. If you want to keep one of the duplicated items, we only need to change the second step a little bit. The third step will not be needed since we can directly extract the first item of the each grouped list:

Extraction (final step)

List<Order> nonDuplicates = maps.entrySet().stream()
       .flatMap(map -> map.getValue().values().stream()
           .map(orders -> orders.get(0)))
       .collect(Collectors.toList());

this will remove all the duplicates. I need to keep one record of it — Krishnanunni P V, Apr 13 '21 at 09:46

score 0 · Answer 3 · answered Apr 13 '21 at 07:47

Use TreeSet to remove duplicated objects

class TestClass
{
    String value1;
    int value2;
    String value3;
}

// here we consider the objects with same value1 and value2 are equal to each other
TreeSet<TestClass> set = new TreeSet<>( (obj1,obj2)->
        Objects.equals(obj1.value1,obj2.value1) && obj1.value2 == obj2.value2 ? 0 :
                obj1.value3.compareTo(obj2.value3)
);

TestClass tc1 = new TestClass();
tc1.value1 = "1";
tc1.value2 = 1;
tc1.value3 = "tc1";
// tc2 has the same value1 and value2 with tc1
TestClass tc2 = new TestClass();
tc2.value1 = "1";
tc2.value2 = 1;
tc2.value3 = "tc2";
TestClass tc3 = new TestClass();
tc3.value1 = "2";
tc3.value2 = 1;
tc3.value3 = "tc3";

set.add(tc1);
set.add(tc2);
set.add(tc3);

for(TestClass tc : set)
{
    System.out.printf("v1 : %s, v2: %s, v3: %s \n",tc.value1,tc.value2,tc.value3);
}

output shows tc2 are not added to TreeSet

v1 : 1, v2: 1, v3: tc1 
v1 : 2, v2: 1, v3: tc3

adarsh · Answer 4 · 2021-04-13T07:58:34.447

One way is to group the objects based on keys (requires correct implementation of equals and hashCode methods for Order):


var t = Arrays.asList(new Order("1", "AA", "Not Completed"), 
                      new Order("1", "AA", "Completed"),
                      new Order("1", "AA", "Completed")
                      new Order("11", "AA", "Not Completed"), 
                      new Order("12", "AB", "Completed"), 
                      new Order("1", "DD", "Completed"),
                      new Order("1", "DD", "Completed"));

Function<Order, String> mapKey = order -> order.id + order.name + "COMPLETED";

// import static java.util.stream.Collectors.*;
var distinctOrders = t.stream().filter(p -> p.status == "Completed")
                               .collect(groupingBy(mapKey, toSet()));

// your distinct orders
distinctOrders.values().forEach(System.out::println);

score 0 · Answer 5 · answered Apr 13 '21 at 09:02

Since you need to collect duplicates only for elements which have status == COMPLETED you could use a map with an appropriate key.

Ideally you'd provide your own key class which takes id, name and status and whose equals() deals status differently based on its value, e.g. like this:

boolean equals(Object other) {
  //check class, id and name first

  //in case of this key any status other than COMPLETE is unqiue and thus orders are not equal
  if( status != COMPLETE ) {
    return false;
  }

  //Key is the key class used for your map
  Key otherOrder = (Key)other;

  //if we reach this, this.status must be COMPLETE so just check other.status 
  return otherOrder.status == COMPLETE;
}

hashCode() should be implemented accordingly although it wouldn't matter if you just use status.hashCode() to build the key hash code. This would lead to collisions of incomplete orders with the same id and value but that could happen anyway and equals() would still make the keys distinct.

Then just put everything into a map:

Map<Key, Order> map = order.stream().collect(Collectors.toMap(
                                         o -> new Key(o), //map order to a new key
                                         o -> o,  //use the order as the value
                                         (l, r) -> l, //merge function to handle duplicates: keep the order already in the map 
                                         LinkedHashMap::new)); //use a LinkedHashMap to keep original order

If you don't want to use a custom key class you could use a list instead (like you're doing it already). Just map the status, e.g. like this lambda for the key mapper:

o -> List.of(o.id, o.name, o.status == COMPLETED ? o.status : new Object())

How does this work? List.of(...) returns an immutable list of type ListN whose equals() method calls equals() on elements at corresponding indices. If id and name are equal the "status" element will be checked. If you "map" any COMPLETED status to just that and any other status to new Object() you will effectively have equal keys for orders with the same id, name and status COMPLETED and unique keys for those with any other status (because Object.equals() uses instance identity only).

Remove duplicates from list based on some property

5 Answers5