0

I have a Dag in air-flow looks like that:

start >> [task1,task2,....task16] >> end

I want to limit the tasks that running simultaneously in this dag to be 4 for example.

I know there is 'max_active_tasks_per_dag' parameters, but it affects on all dags and in my case I want to define it only for my dag. how can I do so? it is even possible?

Dean Taler
  • 737
  • 1
  • 10
  • 25
  • I think you are looking for `concurrency` param. https://stackoverflow.com/questions/56370720/how-to-control-the-parallelism-or-concurrency-of-an-airflow-installation – Emma May 11 '22 at 14:11

1 Answers1

1

If you look at the description of the parameter you will find out that you can configure it per dag:

https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#max-active-tasks-per-dag

max_active_tasks_per_dag New in version 2.2.0.

The maximum number of task instances allowed to run concurrently in each DAG. To calculate the number of tasks that is running concurrently for a DAG, add up the number of running tasks for all DAG runs of the DAG. This is configurable at the DAG level with max_active_tasks, which is defaulted as max_active_tasks_per_dag.

Jarek Potiuk
  • 19,317
  • 2
  • 60
  • 61