8

I have retry logic for tasks and it's not clear how Airflow handles task failures when retries are turned on.

Their documentation just states that on_failure_callback gets triggered when a task fails, but if that task fails and is also marked for retry does that mean that both the on_failure_callback and on_retry_callback would be called?

genhernandez
  • 453
  • 1
  • 5
  • 19

1 Answers1

26

Retry logic/parameters will take place before failure logic/parameters. So if you have a task set to retry twice, it will attempt to run again two times (and thus executing on_retry_callback ) before failing (and then executing on_failure_callback).

An easy way to confirm the sequence that it is executed in is to set your email_on_retry and email_on_failure to True and see the order in which they appear. You can physically confirm that it will retry before failing.

default_args = {
    'owner': 'me',
    'start_date': datetime(2019, 2, 8),
    'email': ['you@work.com'],
    'email_on_failure': True,
    'email_on_retry': True,
    'retries': 1,
    'retry_delay': timedelta(minutes=1)
}
Zack
  • 2,296
  • 20
  • 28
  • thank you! this totally helped. i appreciate the explanation and the steps I can take to confirm! – genhernandez Feb 08 '19 at 21:35
  • @genhernandez no problem! Happy to help. – Zack Feb 08 '19 at 22:32
  • 1
    A separate question to this, if we do not set the retry_delay parameter, but set the retries parameter, what would be the default retry_delay, I have verified that it does work with passing the retry_delay value but do not have an insight on what the default value is.Thanks – Akash Shah Jan 14 '21 at 11:00
  • 1
    @AkashShah `retry_delay: timedelta = timedelta(seconds=300)` according to the docs here https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/index.html#airflow.models.BaseOperator – punkrockpolly Jul 06 '21 at 18:25