Check IEnumerable for items having duplicate properties

Question

How to check if an IEnumerable has two or more items with the same property value ?

For example a class

public class Item
{
    public int Prop1 {get;set;}
    public string Prop2 {get;set;}
}

and then a collection of type IEnumerable<Item>

I need to return false if there are items with duplicate values in Prop1.

digEmAll · Answer 1 · 2016-07-19T06:54:28.273

21

You want to check only for Prop1 right ?

What about:

IEnumerable<Item> items = ...
var noDistinct = items.GroupBy(x => x.Prop1).All(x => x.Count() == 1);
// it returns true if all items have different Prop1, false otherwise

edited Jul 19 '16 at 06:54

answered Jan 04 '11 at 19:30

digEmAll

56,430
9
115
140

1

instead of `noDuplicated`, I would have used `notDistinct` as the variable name and use and `Any` operator that checks if Count is bigger then one. by using `Any` you are not forcing yourself to itterate over the whole enumerable and exit the loop on the first case that matches the `Any` clause – shirbr510 Jul 18 '16 at 15:59
@shirbr510, `All` doesn't force to iterate over the whole enumerable and exits the loop on the first case that **doesn't** match `All` clause. Unlike `Any`, `All` returns true for an empty enumerable, which seems better in this case. More on `All` and `Any` here: https://stackoverflow.com/a/9027666 – axmrnv Feb 10 '20 at 14:04

Eamon Nerbonne · Answer 2 · 2016-03-02T12:05:01.563

19

A short, one-enumeration only solution would be:

public static bool ContainsDuplicates<T>(this IEnumerable<T> list)
    => !list.All(new HashSet<T>().Add);

which could be read as: A list has no duplicates when All items can be Add-ed to a set.

This is conceptually similar to Jake Pearsons solution; however, it leaves out the independant concept of projection; the OP's question would then be solved as:

items.Select(o => o.Prop1).ContainsDuplicates()

edited Mar 02 '16 at 12:05

answered Jan 17 '11 at 11:17

Eamon Nerbonne

47,023
20
101
166

It seems the formatting got messed up in your last edit and the code block is now formatted as plain text. And more importanty this caused the ``-tags to be removed, making the code invalid. I would edit it myself, but since it is only whitespace, and I have nothing else to add, I'm not allowed to... – mflodin Feb 28 '12 at 15:04
thanks, fixed! (weird mistake... I wonder what caused that; looks like some kind of auto-tab conversion) – Eamon Nerbonne Feb 29 '12 at 10:56
@EamonNerbonne n1, also: `Func, bool> containsDuplicates = list => !list.All(new HashSet().Add);` – Mar 01 '16 at 13:08
Don't like the new in the predicate. I know it will be called only once, but I prefer using 2 lines to make it clear... – Maxence Aug 23 '22 at 16:12

score 18 · Accepted Answer · edited Sep 10 '18 at 07:06

18

I think this method will work.

public static bool ContainsDuplicates<T1>(this IEnumerable<T1> source, Func<T1, T2> selector)
{
    var d = new HashSet<T2>();
    foreach(var t in source)
    {
        if(!d.Add(selector(t)))
        {
            return true;
        }
    }
    return false;
}

edited Sep 10 '18 at 07:06

martijnn2008

3,552
5
30
40

answered Jan 04 '11 at 19:22

Jake Pearson

27,069
12
75
95

2

Or just `if (!d.Add(selector(t))) { return false; }` for the loop body. – user7116 Jan 04 '11 at 19:25
Good call, that's a bit faster. – Jake Pearson Jan 04 '11 at 19:28
this is perfect! Just one small think. If collection contains a duplicate, true should be returned and when there is no duplicate, false should be returned :-) – user137348 Jan 04 '11 at 21:29
1

the answer by @digEmAll also only traverses once – Adam Ralph Dec 14 '11 at 14:37
4

The method declaration should be ContainsDuplicates otherwise the code will not compile. – Silwing Aug 26 '13 at 19:18

score 5 · Answer 4 · answered May 26 '14 at 01:08

5

bool x = list.Distinct().SequenceEqual(list);

x is true if list has duplicates.

answered May 26 '14 at 01:08

mnsr

12,337
4
53
79

2

a nifty and readable solution but if I'm not wrong, it will cause multiple enumerations of `list` – shirbr510 Jul 18 '16 at 16:05

user541686 · Answer 5 · 2011-01-04T19:29:16.773

3

Have you tried Enumerable.Distinct(IEnumerable, IEqualityComparer)?

edited Jan 04 '11 at 19:29

answered Jan 04 '11 at 19:23

user541686

205,094
128
528
886

6

This won't tell him if there are duplicate values, it will merely return the distinct values based on the comparer. – user7116 Jan 04 '11 at 21:10
2

Just use `Count()` on both results to see if anything was removed... that was implied. – user541686 Jan 04 '11 at 21:13
3

Some posters may be aware, some would not (and judging from the question, the OP may not be one of them). – user7116 Jan 04 '11 at 21:20

JSBձոգչ · Answer 6 · 2012-09-02T21:03:16.097

3

This could potentially be made for performant, but it's the only correct answer so far.

// Create an enumeration of the distinct values of Prop1
var propertyCollection = objectCollection.Select(o => o.Prop1).Distinct();

// If the property collection has the same number of entries as the object
// collection, then all properties are distinct. Otherwise there are some
// duplicates.
return propertyCollection.Count() == objectCollection.Count();

edited Sep 02 '12 at 21:03

answered Jan 04 '11 at 19:32

JSBձոգչ

40,684
18
101
169

1

Ironically it is NOT a correct answer. Hint: Prop1 needs to be distinct, not Prop2. – cdiggins Sep 02 '12 at 18:56

Jamie Dixon · Answer 7 · 2011-01-04T19:31:16.843

2

You can select the distinct values from the IEnumerable and then check the count against that of the full collection.

Example:

var distinctItemCount = myEnumerable.Select(m => m.Prop1).Distinct().Count();

if(distinctItemCount < myEnumerable.Count())
{
 return false;
}

edited Jan 04 '11 at 19:31

answered Jan 04 '11 at 19:23

Jamie Dixon

53,019
19
125
162

1

Doesn't work. `Distinct` checks for equality, and the objects that the OP is comparing are not necessarily equal. – JSBձոգչ Jan 04 '11 at 19:24
Well then you'd chance `m => m` to `m => m.Prop1` – PostMan Jan 04 '11 at 19:25
I think I updated my code just before you posted your comment. Cheers. – Jamie Dixon Jan 04 '11 at 19:36

score 0 · Answer 8 · answered Apr 27 '16 at 09:55

public static class EnumerableEx
{
    public static IEnumerable<T> GetDuplicates<T>(this IEnumerable<T> source)
    {
        return source.GroupBy(t => t).Where(x => x.Count() > 1).Select(x => x.Key);
    }
}

Personally, I like the neatness of extension methods. If your objects don't require a selector for determining equality, then this works nicely.

score -2 · Answer 9 · edited Oct 22 '13 at 21:18

-2

We can remove duplicate entries by using .Distinct() in ArrayList.

Example:

I have a createdby column in testtable with 5 duplicate entries. I have to get only one row

 ID   Createdby
 ===  ======== 
  1    Reddy
  2    Reddy
  3    Reddy
  4    Reddy

Considering the above table, I need to select only one "Reddy"

DataTable table=new DataTable("MyTable");//Actually I am getting this table data from database

DataColumn col=new DataColumn("Createdby");

var  childrows =  table.AsEnumerable().Select( row => row.Field<object>(col)).Distinct().ToArray();

edited Oct 22 '13 at 21:18

Matthieu

4,605
4
40
60

answered May 29 '12 at 13:56

Rama Subba Reddy M

687
7
14

`DataTables`? `DataColumns`? Question is about knowing if there is a duplicate in an enumerable, not removing them. – c-chavez Apr 19 '22 at 14:17

Check IEnumerable for items having duplicate properties

9 Answers9

Example: