Remove duplicate items from list

Question

I tried following this post but, it doesnt seem to be working for me.

I tried this code:

for bresult in response.css(LIST_SELECTOR):
    NAME_SELECTOR = 'h2 a ::attr(href)'
    yield {
        'name': bresult.css(NAME_SELECTOR).extract_first(),
    }
                                                                                  b_result_list.append(bresult.css(NAME_SELECTOR).extract_first())

    #set b_result_list to SET to remove dups, then change back to LIST
    set(b_result_list)
    list(set(b_result_list))
for brl in b_result_list:
    print("brl: {}".format(brl))

This prints out:

brl: https://facebook.site.com/users/login
brl: https://facebook.site.com/users
brl: https://facebook.site.com/users/login

When I just need:

brl: https://facebook.site.com/users/login
brl: https://facebook.site.com/users

What am I doing wrong here?

Thank you!

score 7 · Accepted Answer · answered Jan 11 '17 at 23:57

7

you are discarding the result when you need to save it ... b_result_list never actually changes... so you are just iterating over the original list. instead save the result of the set operation

b_result_list = list(set(b_result_list))

(note that sets do not preserve order)

answered Jan 11 '17 at 23:57

Joran Beasley

110,522
12
160
179

Wow, silly me. I must be getting tired. Thanks @Joran ! Ill accept as soon as I can – Jshee Jan 12 '17 at 00:01
1

Here, it is not necessary to type-case the `set` back to `list` – Moinuddin Quadri Jan 12 '17 at 00:02
@MoinuddinQuadri that may be true ... but i didnt want to make assumtions – Joran Beasley Jan 12 '17 at 00:05

dawg · Answer 2 · 2017-01-12T01:24:00.943

If you want to maintain order and uniqueify, you can do:

>>> li
['1', '1', '2', '2', '3', '3', '3', '3', '1', '1', '4', '5', '4', '6', '6']
>>> seen=set()
>>> [e for e in li if not (e in seen or seen.add(e))]
['1', '2', '3', '4', '5', '6']

Or, you can use the keys of an OrderedDict:

>>> from collections import OrderedDict
>>> OrderedDict([(k, None) for k in li]).keys()
['1', '2', '3', '4', '5', '6']

But a set alone may substantially change the order of the original list:

>>> list(set(li))
['1', '3', '2', '5', '4', '6']

Remove duplicate items from list

2 Answers2

Linked

Related