How to remove (via linq) duplicates from a List of objects

Question

My main object, has a property which is a List of tags

    [SharedCosmosCollection("shared")]
    public class GlobalPageTemplate : ISharedCosmosEntity
    {
        /// <summary>
        /// Id
        /// </summary>
        [JsonProperty("Id")]
        public string Id { get; set; }

        /// <summary>
        /// Cosmos Entity name
        /// </summary>
        [CosmosPartitionKey]
        public string CosmosEntityName { get; set; }

        /// <summary>
        /// Page name
        /// </summary>
        public string ExtractedPageName { get; set; }

        /// <summary>
        /// Site collection Template Name
        /// </summary>
        public string ExtractedSitecollectionTemplateName { get; set; }

        /// <summary>
        /// GlobalDesignTenantId
        /// </summary>
        public string ExtractedGlobalDesignTenantId { get; set; }

        /// <summary>
        /// Global design tenant site collection url
        /// </summary>
        public string ExtractedGlobalDesigntenantSiteCollectionUrl { get; set; }


        /// <summary>
        /// Page template picture Url
        /// </summary>
        public string PageTemplatePictureUrl { get; set; }

        /// <summary>
        /// Base64 image of the page template
        /// </summary>
        public string Base64Image { get; set; }

        /// <summary>
        /// Name of the template
        /// </summary>
        public string PageTemplateName { get; set; }


        /// <summary>
        /// Page sections
        /// </summary>
        public List<Section> Sections { get; set; }

        /// <summary>
        /// Tags
        /// </summary>
        public List<Tag> Tags { get; set; }
    }

Tag object is here:

 public class Tag : ISharedCosmosEntity
    {
        /// <summary>
        /// Id
        /// </summary>
        [JsonProperty("Id")]
        public string Id { get; set; }
        /// <summary>
        /// Tag name
        /// </summary>
        public string TagName { get; set; }
        /// <summary>
        /// cosmos entity name
        /// </summary>
        [CosmosPartitionKey]
        public string CosmosEntityName { get; set; }
    }

In my WebAPI, from the frontend, I might get duplicate tags,

how do I remove them and leave a clean list of tags before saving?

Answer 1

Can I suggest altering your data structure that stores your tags to a HashSet? If so, you can then do something like this.

A HashSet is an unordered collection of unique elements. It is generally used when we want to prevent duplicate elements from being placed in a collection. The performance of the HashSet is much better in comparison to the list.

Essentially, you supply a custom IEqualityComparer to your HashSet on initialization.

public class TagComparer : IEqualityComparer<Tag>
{
    public bool Equals(Tag x, Tag y)
    {
        return x.Id.Equals(y.Id, StringComparison.InvariantCultureIgnoreCase);
    }

    public int GetHashCode(Tag obj)
    {
        return obj.Id.GetHashCode();
    }
}

And then you can do

HashSet<Tag> Tags = new HashSet<Tag>(new TagComparer());

In general, I always try to use data structures that make sense for the problem at hand. If you know you'll always want this collection to have unique elements, then I suggest you use a HashSet.

If you can't use a HashSet and you want to stick with a list, you can use linq's Distinct method on your Tags list and pass in the TagComparer object from above .

List<Tag> DistinctTagList = Tags.Distict(new TagComparer())

Answer 2

what you are looking for is probably the distict method: https://learn.microsoft.com/en-us/dotnet/api/system.linq.enumerable.distinct?view=netframework-4.8

for that you would also need to write an IEqualityComparer, which can simply compare by property https://learn.microsoft.com/en-us/dotnet/api/system.collections.generic.iequalitycomparer-1?view=netframework-4.8

Then you could call it on your Enumerable:

var distinctTags = Tags.Distict(new TagEqualityComparer)

And the equalityComparer:

class TagEqualityComparer : IEqualityComparer<Tag>
{
    public bool Equals(Tag t1, Tag t2)
    {
        if (t2 == null && t1 == null)
           return true;
        else if (t1 == null || t2 == null)
           return false;
        else if(t1.Id == t2.Id)
            return true;
        else
            return false;
    }

    public int GetHashCode(Tag t)
    {
        // any custom hashingfunction here
    }
}

Answer 3

Using only linq you can do this:

If tags have unique ids:

tags.GroupBy(x => x.Id).Select(x => x.First()).ToList();

If you need to compare all columns:

tags.GroupBy(x => new {x.Id, x.TagName, x.CosmosEntityName}).Select(x => x.First()).ToList();

Answer 4

Not exactly an answer to your question (the other answers are all valid solutions for that), but if for some reason you're looking to actually extract your duplicate objects, such as for debugging, error processing, whatever, I wanted to offer the below.

var duplicates = someList
  .GroupBy(r => r.Id)
  .Where(g => g.Count() > 1)
  .ToList();

Then you have a slightly different way to manage your list from pure distinct

someList = someList.Except(duplicates).ToList();

Which is then a list of keys which had no duplicates.

Answer 5

var applications = from i in db.Users
                               join m in db.Messages on i.Id equals m.fromId
                               where m.toId == userID
                               orderby m.messageId descending
                               select i;

            Cusers = applications.DistinctBy(x => x.Id).ToList();