0

When running Airflow server for the first time after installation, in the DAG list, there is already some dags, such as "example_dash_operator", "example_branch_labels", etc.

Airflow doc says that, to create our own DAGs, we should put in a dags folder, which should be in this location AIRFLOW_HOME/airflow/dags/ (AIRFLOW_HOME is the folder where I install Airflow). I put a sample dag1.py in this folder. But after re-logging in into localhost:8080, I still see only the standard list of DAGS after installation. I don't see dag1.py. I have both the server and the scheduler running with :

airflow webserver --port 8080

airflow scheduler

The full folder structure is as following:

\AIRFLOW_HOME\
      airflow\
           airflow-webserver.pid
           airflow.db
           logs\
           airflow.cfg
           dags\
               dag1.py
           webserver_config.py

This thread here advised to run airflow dags list first. dag1.py does not appear on the list when I run that command. And, after running that, restarting the server and scheduler, the web UI still does not list dag1.py

in airflow.cfg, I have this line defining the dags folder:

dags_folder = /xxxxxx/airflow/dags

where the xxxxxx is the absolute path of AIRFLOW_HOME.

The content of dag1.py is code copied from a tutorial in this Youtube. So I think it is a valid dag.

What am I missing?

Tristan Tran
  • 1,351
  • 1
  • 10
  • 36
  • Example DAGs are coming from [load_examples](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#load-examples) setting. Set to false if you only want your own dags loaded from dags folder. – Hitobat Sep 08 '21 at 21:04

1 Answers1

0

The AIRFLOW_HOME is NOT where you install airflow. The AIRFLOW_HOME is where you aither:

  • have AIRFLOW_HONE environment point to
  • or if you have no AIRFLOW_HOME variable defined when you run airflow, it defaults to "${HOME}/airflow"

See:

https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html?highlight=airflow_home

The first time you run Airflow, it will create a file called airflow.cfg in your $AIRFLOW_HOME directory (~/airflow by default). This file contains Airflow's configuration and you can edit it to change any of the settings. You can also set options with environment variables by using this format: AIRFLOW__{SECTION}__{KEY} (note the double underscores).

Jarek Potiuk
  • 19,317
  • 2
  • 60
  • 61
  • I edited my question to include where AIRFLOW_HOME is. It is my virtual environment locaition where the ```airflow\``` folder is created. If this is the case, what am I doing wrong with the ```dags``` folder? – Tristan Tran Sep 08 '21 at 20:49
  • What's your dag1.py content ? You need it importable, and it needs to create some DAGs when parsed. – Jarek Potiuk Sep 08 '21 at 21:21