4

I am passing some run-time DAG params/config to a PythonOperator in a very similar way to these Airflow docs:

def print_x(x):
    print(f"x is {x}")


with DAG(
    "print_x",
    start_date=pendulum.datetime(2022, 6, 15, tz="UTC"),
    schedule_interval=None,
    catchup=False,
    params={
        "x": Param(42),
    },
) as dag:
    PythonOperator(
        task_id="print_x",
        op_kwargs={
            "x": "{{ params.x }}",
        },
        python_callable=print_x,
    )

But when I manually trigger the DAG I always get the default value (42) regardless of what I put into the "Trigger DAG" dialog box.

What's going wrong here?

Note: In my particular case I'm running Airflow 2.2.2 on AWS Managed Workflows for Apache Airflow (MWAA) but I don't think that should be relevant.

LondonRob
  • 73,083
  • 37
  • 144
  • 201

1 Answers1

5

There's a little discussed param which controls whether config passed when triggering DAGs gets used in the DagRun's tasks:

At the bottom of the Params concepts page it says (the emphasis is mine):

The ability to update params while triggering a DAG depends on the flag core.dag_run_conf_overrides_params. Setting this config to False will effectively turn your default params into constants.

On MWAA it seems like this is set to False by default. You can see whether this is the problem by the message you get at the bottom of the "Trigger DAG" page:

As core.dag_run_conf_overrides_params is set to False, so passing any configuration here won't override task params.

Trigger DAG dialog

LondonRob
  • 73,083
  • 37
  • 144
  • 201
  • In other words, when MWAA states "All Airflow configuration options are using default values." for a given environment, that doesn't mean the configuration parameters are using the default values as specified by the Airflow docs for that version of Airflow! – circld Jun 23 '23 at 00:42
  • Thanks for this. This begs the question: what's the use for this config parameter, why would you be using params at all when this config option is set to false? – Leon Mergen Aug 25 '23 at 04:02