I get this warning when running scrapy :
[scrapy] WARNING: Disabled HttpCacheMiddleware: Unable to find scrapy.cfg file to infer project data dir
Asked
Active
Viewed 304 times
1
Steven Almeroth
- 7,758
- 2
- 50
- 57
Vanddel
- 1,094
- 3
- 13
- 32
1 Answers
1
Where is your scrapy.cfg file? Are you running from the command line? If so, you should be running from the same directory as scrapy.cfg.
What does your project directory look like. It shoud look something like:
.
├── dirbot
│ ├── __init__.py
│ ├── items.py
│ ├── pipelines.py
│ ├── settings.py
│ └── spiders
│ ├── dmoz.py
│ └── __init__.py
├── README.rst
└── scrapy.cfg
UPDATE
If you are using cron, try not to (meaning don't) use the root user. You can add the PATH info right into the crontab: How to get CRON to call in the correct PATHs
PATH=/usr/local/sbin:...:/absolute/path/to/my/scrapy/project
Community
- 1
- 1
Steven Almeroth
- 7,758
- 2
- 50
- 57
-
Project directory is the same as the one you described, the spider is called from a scheduler.php file in the same dir as scrapy.cfg – Vanddel Apr 02 '13 at 12:20
-
How are you calling scrapy? Popen()? who is calling scheduler.php? Apache? It sounds like the user does not have this directory in their Python path. – Steven Almeroth Apr 02 '13 at 12:45
-
a cron job calls scheduler.php which runs the spiders that should be running now. – Vanddel Apr 02 '13 at 12:47
-
What is the cron user? He does not have your directory in his Python path. – Steven Almeroth Apr 02 '13 at 12:51
-
The cron user is root, how do I add the directory to his python path? – Vanddel Apr 02 '13 at 12:56