How to remove duplicate dictionaries from a list in Python?

Question

I have a list of dictionaries sorted by a specific key. Each dictionary contains 32 elements, and there over 4000 dictionaries in the list. I need code to work through the list and return a new list with all duplicates removed.

The methods from these links:

• Removing duplicates in lists
• How do you remove duplicates from a list in whilst preserving order?

Don't help me because a dictionary is unhashable.

Any thoughts? If you need more info, comment and I will add the info.

Edit:

A duplicate dictionary would be any two dictionaries with the same values for list[dictionary][key].

---

Ok, here is a detailed explanation for those who need it.

I have a list of dictionaries like thus:

[ {
    "ID" : "0001",
    "Organization" : "SolarUSA",
    "Matchcode" : "SolarUSA, Something Street, Somewhere State, Whatev Zip",
    "Owner" : "Timothy Black",
   }, {
    "ID" : "0002",
    "Organization" : "SolarUSA",
    "Matchcode" : "SolarUSA, Something Street, Somewhere State, Whatev Zip",
    "Owner" : "Johen Wilheim",
   }, {
    "ID" : "0003",
    "Organization" : "Zapotec",
    "Matchcode" : "Zapotec, Something Street, Somewhere State, Whatev Zip",
    "Owner" : "Simeon Yurrigan",
   } ]

Of this list, the first and second dictionaries are duplicates because their Matchcodes are identical.

Now this list is sorted by the following code:

# sort_by is "Matchcode"
def sort( list_to_be_sorted, sort_by ):
    return sorted(list_to_be_sorted, key=lambda k: k[sort_by])

So I have a neat list of dictionaries sorted by Matchcode. Now I just need to iterate over the list, accessing the list[dictionary][key] and deleting duplicates when two key values match.

Answer 1

Just as you can use a tuple to get a hashable equivalent to a list, you can use a frozenset to get a hashable equivalent to a dict. The only trick is that you need to pass d.items() rather than d to the constructor.

>>> d = {'a': 1, 'b': 2}
>>> s = frozenset(d.items())
>>> hash(s)
-7588994739874264648
>>> dict(s) == d
True

And then you can use your favorite of the solutions you've already seen. Dump them into a set, or use an OrderedSet or the unique_everseen recipe if you need to preserve order, etc. For example:

>>> unique_sets = set(frozenset(d.items()) for d in list_of_dicts)
>>> unique_dicts = [dict(s) for s in unique_sets]

Or, preserving order and using a key value:

>>> sets = (frozenset(d.items()) for d in list_of_dicts)
>>> unique_sets = unique_everseen(sets, key=operator.itemgetter(key))
>>> unique_dicts = [dict(s) for s in unique_sets]

Of course if you have lists or dicts nested within, you will have to convert recursively, just as you would for a list of lists.

Answer 2

Use itertools.groupby() to group your dictionaries by key value, then take the first item from each group.

import itertools

data =[ {
    "ID" : "0001",
    "Organization" : "SolarUSA",
    "Matchcode" : "SolarUSA, Something Street, Somewhere State, Whatev Zip",
    "Owner" : "Timothy Black",
   }, {
    "ID" : "0002",
    "Organization" : "SolarUSA",
    "Matchcode" : "SolarUSA, Something Street, Somewhere State, Whatev Zip",
    "Owner" : "Johen Wilheim",
   }, {
    "ID" : "0003",
    "Organization" : "Zapotec",
    "Matchcode" : "Zapotec, Something Street, Somewhere State, Whatev Zip",
    "Owner" : "Simeon Yurrigan",
   } ]


print [g.next() for k,g in itertools.groupby(data, lambda x: x['Matchcode'])]

Gives the result

[{'Owner': 'Timothy Black',  
  'Organization': 'SolarUSA', 
  'ID': '0001',  
  'Matchcode': 'SolarUSA, Something Street, Somewhere State, Whatev Zip'},

 {'Owner': 'Simeon Yurrigan', 
  'Organization': 'Zapotec', 
  'ID': '0003', 
  'Matchcode':'Zapotec, Something Street, Somewhere State, Whatev Zip'}]

I believe this is what you're looking for.

EDIT: I like the unique_justseen solution better. It's shorter and more descriptive.

Answer 3

This answer is not correct for the now-disambiguated problem.

---

• Do the dicts all have the same keys? If so, write a function like

  the_keys = ["foo", "bar"]
  def as_values(d):
      return tuple(d[k] for k in the_keys)
  
  unique_values = unique_everseen(list_of_dicts, key=as_values)
  

  where unique_everseen is defined at http://docs.python.org/2/library/itertools.html

• If the dicts are not so consistent, use a more general key, such as the FrozenDict I posted to https://stackoverflow.com/a/2704866/192839

Answer 4

So I have a neat list of dictionaries sorted by Matchcode. Now I just need to iterate over the list, accessing the list[dictionary][key] and deleting duplicates when two key values match.

I'm still not entirely sure what this means. It sounds like you're saying they will always be sorted by the same key you want to use to uniquify. If so, you can just use unique_justseen from the itertools recipes, using the same key function you used in the sort, such as itemgetter(key).

Using the sample list_of_dicts from your edited question:

>>> list(unique_justseen(list_of_dicts, key=itemgetter('Matchcode')))
[{'ID': '0001',
  'Matchcode': 'SolarUSA, Something Street, Somewhere State, Whatev Zip',
  'Organization': 'SolarUSA',
  'Owner': 'Timothy Black'},
 {'ID': '0003',
  'Matchcode': 'Zapotec, Something Street, Somewhere State, Whatev Zip',
  'Organization': 'Zapotec',
  'Owner': 'Simeon Yurrigan'}]

---

If they're sorted by a different key to the one we're uniquifying on, then the fact that they're sorted isn't relevant at all, and unique_justseen won't work:

>>> list_of_dicts.sort(key=itemgetter('Owner'))
>>> list(unique_justseen(list_of_dicts, key=itemgetter('Matchcode')))
[{'ID': '0002',
  'Matchcode': 'SolarUSA, Something Street, Somewhere State, Whatev Zip',
  'Organization': 'SolarUSA',
  'Owner': 'Johen Wilheim'},
 {'ID': '0003',
  'Matchcode': 'Zapotec, Something Street, Somewhere State, Whatev Zip',
  'Organization': 'Zapotec',
  'Owner': 'Simeon Yurrigan'},
 {'ID': '0001',
  'Matchcode': 'SolarUSA, Something Street, Somewhere State, Whatev Zip',
  'Organization': 'SolarUSA',
  'Owner': 'Timothy Black'}]

But then you just have to use the unique_everseen recipe instead:

>>> list_of_dicts.sort(key=itemgetter('Owner'))
>>> list(unique_everseen(list_of_dicts, key=itemgetter('Matchcode')))
[{'ID': '0002',
  'Matchcode': 'SolarUSA, Something Street, Somewhere State, Whatev Zip',
  'Organization': 'SolarUSA',
  'Owner': 'Johen Wilheim'},
 {'ID': '0003',
  'Matchcode': 'Zapotec, Something Street, Somewhere State, Whatev Zip',
  'Organization': 'Zapotec',
  'Owner': 'Simeon Yurrigan'}]

(Of course this time we've got 0002 instead of 0001, because after sorting on Owner it's now the first value for its Matchcode instead of the second.)

---

The fact that dictionaries aren't hashable isn't relevant here, because the recipes are just storing the result of the key function in their sets, so as long as the values stored at key key are hashable, everything is fine.

Answer 5

Now that we can see that two dictionaries are duplicates if a particular key matches, the problem is very simple. Just iterate over the dictionaries; keep track of the keys you've seen, and finally make a new list out of the unique ones.

import collections
def get_unique_items(list_of_dicts, key="Matchcode"):
    # Count how many times each key occurs.
    key_count = collections.defaultdict(lambda: 0)
    for d in list_of_dicts:
        key_count[d[key]] += 1

    # Now return a list of only those dicts with a unique key.
    return [d for d in list_of_dicts if key_count[d[key]] == 1]

Note that I use a defaultdict here to count occurrences of each key (there are other ways to do this but I think this is the cleanest, personally). The reason I didn't use a set to track "visited" keys is that you'll get one copy of every key in the list, including the duplicated ones. That means you would have to keep a second set, to track the keys that are truly duplicated (those that are already in the "visited" keys when you encounter them), so you don't include them.

On the other hand, if what you want is just to take the first dictionary you see with each given key, whether there's a duplicate or not later on, a set approach would work fine, like in Mike Graham's second answer.

Answer 6

seen_values = set()
without_duplicates = []
for d in list_of_dicts:
    value = d[key]
    if value not in seen_values:
        without_duplicates.append(d)
        seen_values.add(value)

Answer 7

Basically you need something like no_dup(checked_val_extrator, list) where no_dup may look like:

def no_dup(extractor, lst):
   "keeps only first elements encountered for any particular extracted value using =="
   known = set()
   res = []
   for item in lst:
     if extractor(item) in known: continue
     known.add(extractor(item))
     res.append(item)
   return res

Answer 8

I'm not 100% clear on exactly what you're trying to achieve, but:

Remove all duplicate dictionary entries

So long as you don't mind combining all the dictionaries,

import itertools
dict(itertools.chain(*map(lambda x: x.items(), list_of_dictionaries)))