How to eliminate duplicate elements in a list using python?

Question

I have a python script which parses a .txt file and generates a list as follows:

['test=testTC101', 'test=testTC101', 'test=testTC102', 'test=testTC102', 'test=testTC103', 'test=testTC103', 'test=testTC104', 'test=testTC104', 'test=testTC105', 'test=testTC105', 'test=testTC106', 'test=testTC106', 'test=testTC107', 'test=testTC107']

I need to eliminate the duplicates. How can I achieve that?

duplicate: http://stackoverflow.com/q/7961363/1286571, or: http://stackoverflow.com/q/480214/1286571 — ForeverWintr, Sep 24 '13 at 20:45
http://stackoverflow.com/questions/12897374/get-unique-values-from-a-list-in-python — agentp, Sep 24 '13 at 20:47

score 3 · Answer 1 · 2013-09-24T20:55:12.630

Just use set:

>>> x = ['test=testTC101', 'test=testTC101', 'test=testTC102', 'test=testTC102', 
...      'test=testTC103', 'test=testTC103', 'test=testTC104', 'test=testTC104', 
...      'test=testTC105', 'test=testTC105', 'test=testTC106', 'test=testTC106', 
...      'test=testTC107', 'test=testTC107']
>>> set(x)
set(['test=testTC101', 'test=testTC103', 'test=testTC102', 'test=testTC105', 
'test=testTC104', 'test=testTC107', 'test=testTC106'])
>>>

You can then always make it back into a list using list:

>>> list(set(x))
['test=testTC101', 'test=testTC103', 'test=testTC102', 'test=testTC105', 
'test=testTC104', 'test=testTC107', 'test=testTC106']
>>> # You can also use sorted to order the items
>>> sorted(list(set(x)))
['test=testTC101', 'test=testTC102', 'test=testTC103', 'test=testTC104', 
'test=testTC105', 'test=testTC106', 'test=testTC107']
>>>

Sets are guaranteed to not have duplicates.

score 1 · Answer 2 · answered Sep 24 '13 at 20:58

Should you happen to require that order is preserved:

def best_case_iteruniq(iterable, key=None):
    memo_hash = set()
    memo_else = list()

    add_hash = memo_hash.add
    add_else = memo_else.append

    if key is None:
        def predicate(item):
            try:
                hash(item)

            except TypeError:
                if item in memo_else:
                    return False

                add_else(item)
                return True

            else:
                if item in memo_hash:
                    return False

                add_hash(item)
                return True


    else:
        def predicate(actual_item):
            item = key(actual_item)

            try:
                hash(item)

            except TypeError:
                if item in memo_else:
                    return False

                add_else(item)
                return True

            else:
                if item in memo_hash:
                    return False

                add_hash(item)
                return True

    return filter(predicate, iterable)

This is a lil' tool I made some time ago that allows you to uniquify lists of both hashable and non-hashable elements in the minimum possible time (assuming that there are no totally-ordered non-hashable elements), and it also allows a key to uniquify them with.

It's almost definitely overkill for this scenario, though. But it's there, and it's free.

How to eliminate duplicate elements in a list using python?

2 Answers2