5

I'm using Ubuntu 14.04 x64, I've followed the Scrapy docs to install the package using pip:

pip install scrapy

Then I followed the initialization of the example project and try to execute the example spider:

scrapy crawl example

and I get this error:

2015-02-23 10:23:42+0100 [scrapy] INFO: Scrapy 0.14.4 started (bot: example)
2015-02-23 10:23:42+0100 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, MemoryUsage, SpiderState
Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/usr/lib/python2.7/dist-packages/scrapy/cmdline.py", line 132, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/usr/lib/python2.7/dist-packages/scrapy/cmdline.py", line 97, in _run_print_help
    func(*a, **kw)
  File "/usr/lib/python2.7/dist-packages/scrapy/cmdline.py", line 139, in _run_command
    cmd.run(args, opts)
  File "/usr/lib/python2.7/dist-packages/scrapy/commands/crawl.py", line 43, in run
    spider = self.crawler.spiders.create(spname, **opts.spargs)
  File "/usr/lib/python2.7/dist-packages/scrapy/command.py", line 34, in crawler
    self._crawler.configure()
  File "/usr/lib/python2.7/dist-packages/scrapy/crawler.py", line 36, in configure
    self.spiders = spman_cls.from_crawler(self)
  File "/usr/lib/python2.7/dist-packages/scrapy/spidermanager.py", line 37, in from_crawler
    return cls.from_settings(crawler.settings)
  File "/usr/lib/python2.7/dist-packages/scrapy/spidermanager.py", line 33, in from_settings
    return cls(settings.getlist('SPIDER_MODULES'))
  File "/usr/lib/python2.7/dist-packages/scrapy/spidermanager.py", line 23, in __init__
    for module in walk_modules(name):
  File "/usr/lib/python2.7/dist-packages/scrapy/utils/misc.py", line 65, in walk_modules
    submod = __import__(fullpath, {}, {}, [''])
  File "/home/alvaro/vagrantenvs/example/example/spiders/example_spider.py", line 3, in <module>
    class ExampleSpider(scrapy.Spider):
AttributeError: 'module' object has no attribute 'Spider'

I checked with pip freeze | grep 'Scrapy' to check the installed version of Scrapy and I got:

Scrapy==0.24.4

but as you can see in the traceback error:

2015-02-23 10:23:42+0100 [scrapy] INFO: Scrapy 0.14.4 started (bot: example)
  1. I tried to uninstall and reinstall
  2. I tried also to update pip, uninstall scrapy and reinstall
  3. I tried to install it using apt-get
  • Why If I installed the latest version (0.24.4) it seems it's executing 0.14.4 ?
  • Is this issue related with Scrapy, Pip or both ?

Edit:

Here it is the file example_spider.py:

import scrapy

class ExampleSpider(scrapy.Spider):
    name = "example"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        filename = response.url.split("/")[-2]
        with open(filename, 'wb') as f:
            f.write(response.body)

If I execute scrapy version as @aberna sugested I got:

Scrapy 0.14.4
Community
  • 1
  • 1
AlvaroAV
  • 10,335
  • 12
  • 60
  • 91

1 Answers1

6

I had a similar issue back then and It was because the Ubuntu package was messing with the pip package.

When I installed Scrapy using pip and try to run it, scrapy says that the package was missing some dependencies and then I installed Scrapy using apt-get and there is where I screwed up with the package version.

I recommend you to remove both packages from Ubuntu and Pip and then reinstall only using pip:

apt-get remove --purge python-scrapy
pip uninstall scrapy

After those actions finished you can install it again using pip and then if you do scrapy version you should get as response the newest version, in your case it should be 0.24.4.

EDIT

You can read this in Scrapy Documentation:

python-scrapy is a different package provided by official debian repositories, it’s very outdated and it isn’t supported by Scrapy team.

To install Scrapy in Ubuntu they say:

  1. Import the GPG key used to sign Scrapy packages into APT keyring:

    sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 627220E7

  2. Create /etc/apt/sources.list.d/scrapy.list file using the following command:

    echo 'deb http://archive.scrapy.org/ubuntu scrapy main' | sudo tee /etc/apt/sources.list.d/scrapy.list

3.Update package lists and install the scrapy-0.24 package:

sudo apt-get update && sudo apt-get install scrapy-0.24

Check Here: Scrapy on Ubuntu

  • 1
    Thanks ! I had scrapy package installed, after uninstall it with apt and pip and reinstall with pip it works perfect ! – AlvaroAV Feb 27 '15 at 07:46
  • An absolute savior! I was getting such crazy errors due to the version problems on my Ubuntu and this reply saved me big time. Cheers. – AmirHd Jul 10 '15 at 06:14