Having Trouble Removing Duplicates From A List

Question

I have a list of like 3000 items, and I think there are a ton of duplicates. So to make my program run faster, I want to delete them. This is the code along with my function, which when I run the program the text file is empty.

List<string> allCombos = new List<string>();
        string here = RemoveTheSame(allCombos, allCombos);
        System.IO.File.WriteAllText(@"C:\Hodag Image Storage Folder\creo.txt", here);

//////////////////////////////

        private string RemoveTheSame(List<String> lista, List<String> listb)
    {
        List<String> newList = lista;

        newList.AddRange(lista);
        for(int i = 0; i < lista.Count; i++)
        {
            int a = 0;
            for(int ii = 0; ii < listb.Count; ii++)
            {
                if(lista[i] == listb[ii])
                {
                    a += 1;
                    if(a >= 2)
                    {
                        newList.RemoveAt(ii);
                        a = 0;
                    }
                }
            }
        }

        string wat = "";
        for(int i = 0; i < newList.Count; i++)
        {
            wat += newList[i] + " ";
        }


        return wat;
    }

Typo? `string here = RemoveTheSame(allCombos, allCombos);` looks **strange**: you, probably want to pass **different** collections into the method - `string here = RemoveTheSame(allCombos, rerefenceCombos);` — Dmitry Bychenko, Jun 25 '18 at 14:39
Since you're comparing the same list twice, all items will be duplicated right? That means you *should* have an empty output. — DavidG, Jun 25 '18 at 14:40

Rahul · Answer 1 · 2018-06-25T14:42:06.513

1

Not sure, why can't you use Linq Distinct() method, which will retrieve only the unique items from the input list

var distinctItems = allCombos.Distinct().ToList();

edited Jun 25 '18 at 14:42

answered Jun 25 '18 at 14:37

Rahul

76,197
13
71
125

Dmitry Bychenko · Accepted Answer · 2018-06-25T15:31:59.563

As far as I can see from RemoveTheSame(List<String> lista, List<String> listb) you can have three possible cases:

You have two lists: allCombos and referenceList you want to return all the items from allCombos which are not in referenceList:

  allCombos:     [1, 2, 3, 1, 4, 5, 5]
  referenceList: [1, 2, 4]
  result:        [3, 5, 5] // 1, 2, 4 removed

Solution:

   var hashSet = new hashSet(referenceList); 

   allCombos.RemoveAll(item => hashSet.Contains(item));

You have two lists: allCombos and referenceList you want to return all the items from allCombos which are not in referenceList; all duplicates from allCombos should be removed as well:

  allCombos:     [1, 2, 3, 1, 4, 5, 5]
  referenceList: [1, 2, 4]
  result:        [3, 5]    // duplicate - second "5" removed

Solution:

   allCombos = allCombos
     .Except(referenceList)
     .ToList();

You have a single allCombos list only, and want to remove duplicates from it:

   allCombos:     [1, 2, 3, 1, 4, 5, 5]
   result:        [1, 2, 3, 4, 5] // second 1 and 5 are removed

Solution:

   allCombos = allCombos
     .Distinct()
     .ToList();

Having Trouble Removing Duplicates From A List

2 Answers2