1

I have an Airflow 2.0 DAG defined with a start_date in the past, catchup = True and max_active_runs = 5. The goal is to process data from the start_date with up to five DAG Runs executing in parallel.

Certain tasks are defined with depends_on_past = True, however I observe that they do sometimes execute in parallel and it seems depends_on_past is not respected.

Why is this ? Is it possible to process historical data with parallel DAG Runs, with either catchup or backfill, and prevent certain tasks in the DAG from executing in parallel ?

Phil
  • 598
  • 1
  • 9
  • 21

2 Answers2

2

Why is this?

According to documentation this should not happen. Refer to this answer.

Is it possible to process historical data with parallel DAG Runs, with either catchup or backfill, and prevent certain tasks in the DAG from executing in parallel?

You would benefit from the pools. Pools allow you to control parallelism on tasks of your choice. It is referred in this answer as well.

P.S. Would have suggested in comments but I can't.

AnkurSaxena
  • 825
  • 7
  • 11
1

My bad. What I omitted is that I have a short-circuit operator in my DAG. If a Dag Run short circuits and tasks are skipped they treated as success, which triggers downstream tasks that depend on past.

This makes sense - if a task if skipped and not treated as success, future tasks would not execute.

Phil
  • 598
  • 1
  • 9
  • 21