Removing duplicates within list without creating a new list while preserving order

Question

Input:

glist = [1,4,2,5,2,2]

Expected output:

[1,4,2,5]

As you can see, I just want to remove the duplicates, the order of elements I want it to be same

Usual way to remove duplicates:

def remove(glist):
    result = []
    for ele in glist:
        if ele not in result:
            result.append(ele)
    return result

However, this code will create a new list, which is result. I just want to remove the duplicates, WITHOUT creating new list/set. Meaning I want return glist as show below My python: IDLE 3.3 Thanks guys!!!

Presentation of expected code:

def remove(glist):
    '''code'''
    return glist

@karthikr: that will violate the OP's wish for the original order to be preserved. — DSM, Apr 08 '14 at 17:11
It's not clear what "without creating [a] new list" means. Are you just looking for `glist[:] = result` instead of `return result`? (Note that there are more efficient ways to remove duplicates, esp. if your values are hashable, see [here](http://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-in-python-whilst-preserving-order?rq=1)). — DSM, Apr 08 '14 at 17:17
@DSM Hi, thanks for the comment, I have edited the question, I am not looking for `glist[:] = result` or `return result`, I want to return `glist` as it is the original list. — shawnngtq, Apr 08 '14 at 17:34

devnull · Answer 1 · 2014-04-08T17:22:34.160

2

You could use OrderedDict:

>>> glist = [1,4,2,5,2,2]
>>> from collections import OrderedDict
>>> glist = list(OrderedDict.fromkeys(glist))
>>> glist
[1, 4, 2, 5]
>>>

edited Apr 08 '14 at 17:22

answered Apr 08 '14 at 17:15

devnull

118,548
33
236
227

Steve Jessop · Accepted Answer · 2014-04-08T18:23:12.010

Assuming that the contents are hashable, you can do it this way without creating any list object in your code:

def remove(glist):
    seen = set()
    def unseen():
        for val in glist:
            if not val in seen:
                yield val
                seen.add(val)
    glist[:] = unseen()
    return glist # if you must

It's a bit of a cheat, since in effect glist[:] = unseen() does construct a list from the output of the generator function unseen() and then (efficiently) transfer its contents to glist. It may not actually construct a list object, but it uses about the same amount of memory as if it did, because the old contents of glist of course aren't freed until the new array is completed.

If the contents aren't hashable, or if you've plenty of time but no memory to spare, then I suppose you'll have to do it the slow way:

idx = 0
while idx < len(glist):
    if glist[idx] in glist[0:idx]:
        del glist[idx]
    else:
        idx += 1

score -1 · Answer 3 · answered Apr 08 '14 at 17:37

-1

A simpler version of Alex's answer (which is perfectly valid) would be:

glist = [1,4,2,5]
glist = set(glist)
print glist

A set cannot have duplicates, so making a list into a set removes all duplicates.

answered Apr 08 '14 at 17:37

kttr

414
1
3
11

This doesn't preserve order. – anon582847382 Apr 08 '14 at 17:48
@Ajax: Thanks for the reply, but I need to preserve the order, and I think `set` is consider a new list/set as I need to use the original list which is `glist` – shawnngtq Apr 08 '14 at 17:49

Removing duplicates within list without creating a new list while preserving order

3 Answers3