26

According to the official Airflow docs, The task instances directly upstream from the task need to be in a success state. Also, if you have set depends_on_past=True, the previous task instance needs to have succeeded (except if it is the first run for that task).

As all know, the task is kind of 'instantiated & parameteriazed' operator.

Now this is what confuse me. For example:

DAG: {op_1} -> {op_2} -> {op_3}

{op_2} is a simple PythonOperator that takes 1 parameter from {op_1} and do stuff;

To my understanding, op_2(param_1) & op_2(param_2) are considered as 2 different tasks.

Given depends_on_past is set to True, then:

  1. If op_2(param_1) is still running; can op_2(param_2) be run?
  2. If op_2(param_1) fails in the previous run; can op_2(param_1) be run in the current run?
Meghdeep Ray
  • 5,262
  • 4
  • 34
  • 58
WeiHao
  • 732
  • 1
  • 7
  • 9

1 Answers1

35

From the official docs for trigger rules:

depends_on_past (boolean) when set to True, keeps a task from getting triggered if the previous schedule for the task hasn’t succeeded.

So unless a previous run of your DAG has failed, the depends_on_past should not be a factor, it will not affect the current run at all if the previous run executed the tasks successfully.

Meghdeep Ray
  • 5,262
  • 4
  • 34
  • 58
  • 2
    one point to note here is that the task remains in None state (not scheduled) when the task_instance for which depends_on_past is true and the task in previous schedule of the *scheduled dags* (schedule_interval is not None) is not successful. Depends on Past does not work on triggred DAGs – Tameem Jun 28 '19 at 12:56
  • @Tameem "Depends on Past does not work on triggred DAGs" - is that true though? I am manually triggering DAGs and tasks are in a none state with the task instance saying "depends_on_past is true for this task's DAG, but the previous task instance has not run yet.". This is an externally triggered dag run. Appreciate your thoughts since i dont see that mentioned in the docs - despite it making a lot of sense! – theStud54 Oct 16 '19 at 12:56
  • @Tameem "Depends on Past does not work on triggred DAGs" - is there official documentation on this? – Abhilash Kishore Mar 19 '20 at 02:44
  • 2
    @theStud54 you did make a lot of sense... I rechecked the functionality on a simple DAG and it does work as expected. "@Abhilask Kishore" I had observed this when I was working on 1.9.0 but just to be sure before I post this comment, I executed the DAG with depends_on_past and my previous observation was incorrect. – Tameem Mar 27 '20 at 11:48
  • This saved me. I incorrectly thought `depends_on_past` only referred to the previous task success. – user1893354 Jul 26 '20 at 23:40