How can i remove overlap in list?

Question

I made example python list.

list_1 = [1,3,2,2,3,4,5,1]
print(list_1)

[1, 3, 2, 2, 3, 4, 5, 1]

To remove overlap, i tried to use set().

print(set(list_1))

{1, 2, 3, 4, 5}

but i want make

[1,3,2,4,5]

I want remove overlap in list, but i also want order not to be changed.

How can i do that?

Possible duplicate of [How do you remove duplicates from a list in Python whilst preserving order?](http://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-in-python-whilst-preserving-order) — Paul Rooney, Aug 04 '16 at 11:14

score 1 · Answer 1 · edited May 23 '17 at 12:32

You could (ab)use collections.OrderedDict to get just the unique elements in order:

>>> list_1 = [1,3,2,2,3,4,5,1]
>>> from collections import OrderedDict
>>> list(OrderedDict((x, None) for x in list_1))
[1, 3, 2, 4, 5]

Alternatively, you could use a list comprehension with an additional set of already seen items. However, list comprehensions with side-effects are not considered best style.

>>> list_1 = [1,3,2,2,3,4,5,1]
>>> seen = set()
>>> [x for x in list_1 if not (x in seen or seen.add(x))]
[1, 3, 2, 4, 5]

The condition not (x in seen or seen.add(x)) works like this: If x in seen, the or is true and thus the not is false, but if x not in seen, then the first part of the or is false, and thus the second part is executed and returns None, which is then converted to True by not.

Thanks for very kind explanation. – touchingtwist Aug 05 '16 at 04:22 — touchingtwist, Aug 05 '16 at 04:22

score 1 · Answer 2 · answered Aug 04 '16 at 12:13

I would probably do something like this:

seen = set()
result
for i in the_list:
  if i not in seen:
    seen.add(i)
    result.append(i)

I use seen for fast lookups, still unknown time complexity from the result.append method call.

It's possible to do this as a list comprehension as well, at the cost of mixing functional and imperative code in a horrible mish-mash:

seen = set()
def process(e):
  seen.add(e)
  return e
[process(element) for element in the_list if element not in seen]

But! This relies on side-effects in the mapping, so it's less than ideal.

score 1 · Accepted Answer · answered Aug 04 '16 at 13:24

1

You can use list-comprehension with filter (initialize the empty list first, ignore the resulting list)

list_u = []
[list_u.append(v) for v in list_1 if v not in list_u]

answered Aug 04 '16 at 13:24

DurgaDatta

3,952
4
27
34

score 0 · Answer 4 · edited May 23 '17 at 12:32

0

To have distinct values in your list, just use the following snippet:

l = [1,3,2,2,3,4,5,1]
result = list()
map(lambda x: not x in result and result.append(x), l)
print(result) #[1,3,2,4,5]

As you already found out, set() doesn't preserve the order of your sequence. The basic idea is taken from here.

As tobias_k mentioned in the comment, this solution won't work with Python 3. If you use Python 3, you may just iterate through your list with a simple for loop (tobias_k's idea) or use libraries like OrderedDict:

from collections import OrderedDict

list(OrderedDict.fromkeys([1,3,2,2,3,4,5,1]))

edited May 23 '17 at 12:32

Community

1
1

answered Aug 04 '16 at 11:10

oopbase

11,157
12
40
59

IMHO using a plain old `for` loop would be more pythonic than using `map` for side-effects. Also, this won't work in Python 3. – tobias_k Aug 04 '16 at 11:12
@tobias_k Thanks for the comment mate. I agree with you. This isn't the pythonic way to go. But as I'm seeing right now, you also thought of the OrderedDict as a solution :-). – oopbase Aug 04 '16 at 11:18

score 0 · Answer 5 · answered Aug 04 '16 at 11:20

Try This You can do this using reversing a list.without using any python libraries.

>>> list_1 = [1,3,2,2,3,4,5,1]
>>> list_1.reverse()
>>> for i in list_1:
...     if(list_1.count(i)>1):
...        list_1.remove(i)
... 
>>> list_1.reverse()
>>> list_1
[1, 3, 2, 4, 5]

How can i remove overlap in list?

5 Answers5