1

When I try to run my scraper in Django I get the following error.

This happened when I tried to put the scraped data into the database using the built-in Django models.

traceback and error:

/usr/bin/python3.6 /home/xxxxxx/Desktop/project/teo/movierater/scrap.py
Traceback (most recent call last):
  File "/home/xxxxxx/Desktop/project/teo/movierater/scrap.py", line 7, in <module>
    from teo.movierater.api.models import *
  File "/home/xxxxxx/Desktop/project/teo/movierater/api/models.py", line 3, in <module>
    class Author(models.Model):
  File "/usr/local/lib/python3.6/dist-packages/django/db/models/base.py", line 103, in __new__
    app_config = apps.get_containing_app_config(module)
  File "/usr/local/lib/python3.6/dist-packages/django/apps/registry.py", line 252, in get_containing_app_config
    self.check_apps_ready()
  File "/usr/local/lib/python3.6/dist-packages/django/apps/registry.py", line 134, in check_apps_ready
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.6/dist-packages/django/conf/__init__.py", line 79, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.6/dist-packages/django/conf/__init__.py", line 64, in _setup

    % (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

I put it in bin/activate:

export DJANGO_SETTINGS_MODULE=mysite.settings

Here is the code snippet from scraper.py:

import django
import os
import requests
from bs4 import BeautifulSoup as bs
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from teo.movierater.api.models import *


os.environ['DJANGO_SETTINGS_MODULE'] = 'movierater.settings'
django.setup()

When I try import models below django.setup():

import django
import os
import requests
from bs4 import BeautifulSoup as bs
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

os.environ['DJANGO_SETTINGS_MODULE'] = 'movierater.settings'
django.setup()

from teo.movierater.api.models import *

Error:

ModuleNotFoundError: No module named 'movierater'

My project structure and the settings:

enter image description here

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',
    'movierater.api',
]

wsgi.py file:

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'movierater.settings')

application = get_wsgi_application()
tbone
  • 1,148
  • 5
  • 19
  • 35

1 Answers1

1

I can see a couple of possible issues here.

  1. use movierater.settings as DJANGO_SETTINGS_MODULE value, i.e. export DJANGO_SETTINGS_MODULE=movierater.settings

  2. how do you start your app, did you try python manage.py shell?

  3. if you you want to start your scraper as a script (e.g. python manage.py scraper) then the best way to do it is to write a custom command; take a look at use django: from "python manage.py shell" to python script

Dušan Maďar
  • 9,269
  • 5
  • 49
  • 64
  • where should I attach : export DJANGO_SETTINGS_MODULE=movierater.settings – tbone Jul 24 '19 at 19:14
  • It's an environment variable, run `export DJANGO_SETTINGS_MODULE=movierater.settings` in your shell (https://www.digitalocean.com/community/tutorials/how-to-read-and-set-environmental-and-shell-variables-on-a-linux-vps) – Dušan Maďar Jul 24 '19 at 19:16
  • This command export DJANGO_SETTINGS_MODULE=movierater.settings is not valid. It produces a syntax error in the shell. – ionecum Nov 10 '21 at 16:47