1

I have the following class:

class PitchforkTracks(scrapy.Spider):
    name = "pitchfork_tracks"
    allowed_domains = ["pitchfork.com"]
    start_urls = [
                    "http://pitchfork.com/reviews/best/tracks/?page=1",
                    "http://pitchfork.com/reviews/best/tracks/?page=2",
                    "http://pitchfork.com/reviews/best/tracks/?page=3",
                    "http://pitchfork.com/reviews/best/tracks/?page=4",
                    "http://pitchfork.com/reviews/best/tracks/?page=5",
    ]
    def parse(self, response):

        for sel in response.xpath('//div[@class="track-details"]/div[@class="row"]'):
            item = PitchforkItem()
            item['artist'] = sel.xpath('.//li/text()').extract_first()  
            item['track'] = sel.xpath('.//h2[@class="title"]/text()').extract_first()  
            yield item

scraping this item:

<h2 class="title" data-reactid="...>“Colours”</h2>

results, however, print like this:

{'artist': u'The Avalanches', 'track': u'\u201cColours\u201d'}

where and how do I strip out the quotes, i.e, \u201c and \u201d?

Marios
  • 26,333
  • 8
  • 32
  • 52
8-Bit Borges
  • 9,643
  • 29
  • 101
  • 198
  • have you tried http://stackoverflow.com/questions/15321138/removing-unicode-u2026-like-characters-in-a-string-in-python2-7 ? – Ben Sep 30 '16 at 01:19
  • @Ben if I write: `item['track'] = item['track'].decode('unicode_escape').encode('ascii','ignore')`I get this traceback: `UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 0: ordinal not in range(128)` – 8-Bit Borges Sep 30 '16 at 01:33

1 Answers1

2

Inside parse(self, response), add:

item['track'] = sel.xpath('.//h2[@class="title"]/text()').extract_first().strip(u'\u201c\u201d')
8-Bit Borges
  • 9,643
  • 29
  • 101
  • 198