Apache Airflow Best Practices: DAG Design, Task Reliability, and Production Operations

Apache Airflow is the most widely deployed workflow orchestration platform in data engineering. It is also one of the most commonly misused. Airflow's flexibility — you can express almost any pipeline as a DAG in Python — is both its strength and the source of most of its operational problems. Poorly designed Airflow environments degrade over time: DAGs grow unmaintainable, the scheduler becomes slow, tasks fail silently, and the team loses confidence in the system.

This guide covers the practices that separate production-grade Airflow environments from fragile ones.

DAG Design Principles

**DAGs should be idempotent.** Running the same DAG for the same execution date multiple times should produce the same result as running it once. This requires that every task writes data in a way that handles re-runs: upsert or merge rather than append, explicit partition overwrite rather than incremental append to an existing partition, and idempotent file writes to object storage.

The most common failure to idempotency: a task that appends to a table on every run. A failed run that partially completes, followed by a full re-run, produces duplicates. Upsert on a natural key or partition overwrite avoids this.

**Keep DAGs focused.** A DAG that does everything — ingests from 5 sources, runs 20 transformations, exports to 3 downstream systems — is difficult to debug, difficult to schedule (all steps are coupled), and difficult to maintain. Design DAGs around a cohesive unit of work: one source system's ingestion, one domain's transformation, one export pipeline. Cross-DAG dependencies are managed with TriggerDagRunOperator or Airflow datasets.

**Separate concerns between DAGs and task logic.** Task logic (the actual SQL, Python transformation, or API call) should live in importable modules or external files, not inline in the DAG file. A DAG file should be readable as a description of the pipeline structure — task dependencies, scheduling, and retry configuration — not a blob of business logic. Heavy logic in DAG files is unmaintainable and creates DAG parse time problems as the code grows.

**Use descriptive task IDs.** Task IDs are visible in the Airflow UI, in log file paths, and in metadata queries. A task named extract is not useful in a DAG with 15 tasks; extract_salesforce_opportunities_v2 is. Name tasks for what they do, not just their type.

Parameterisation and Dynamic DAGs

**Use Airflow Variables and Connections for configuration.** Hardcoded connection strings, API keys, and configuration values in DAG files are a security risk and a maintenance burden. Store them in Airflow Variables (for configuration values) and Connections (for database/API credentials). DAG code reads them at runtime:

from airflow.models import Variable

snowflake_account = Variable.get("snowflake_account")

**Jinja templating for execution date.** Airflow's templating engine exposes the execution date and other run context in operator parameters:

extract_task = PythonOperator(

task_id="extract",

python_callable=extract_data,

op_kwargs={"execution_date": "{{ ds }}"}, # YYYY-MM-DD string

)

Use execution date parameterisation rather than hardcoded dates or Python's datetime.now(). This ensures that backfill runs (re-running historical execution dates) use the correct date context rather than today's date.

**Dynamic DAG generation.** For pipelines that follow the same pattern across many sources (e.g., extracting from 20 different APIs with the same structure), generate DAGs programmatically from a configuration file rather than writing 20 nearly identical DAG files:

for source_config in config["sources"]:

dag_id = f"extract_{source_config['name']}"

globals()[dag_id] = create_dag(source_config)

Be cautious with dynamic DAG generation — Airflow parses DAG files on every scheduler heartbeat, and expensive code in the top-level module scope slows down the scheduler. Keep dynamic generation logic fast and avoid external calls (database queries, API calls) during DAG file import.

Task Reliability

**Configure retries appropriately.** Most Airflow tasks should have retries configured. A transient API timeout, a temporary warehouse connection failure, or a brief network interruption will cause a task to fail without retries configured. A task that calls an external API should have 2–3 retries with exponential backoff:

extract_task = PythonOperator(

...

retries=3,

retry_delay=timedelta(minutes=5),

retry_exponential_backoff=True,

)

Do not configure retries for tasks where re-running would cause harm — tasks that send notifications, trigger external actions, or write to systems where idempotency is not guaranteed.

**Use task timeouts.** A task without a timeout can hang indefinitely, blocking downstream tasks and consuming executor slots. Set execution_timeout on all tasks that interact with external systems:

extract_task = PythonOperator(

...

execution_timeout=timedelta(hours=2),

)

**Catchup and backfill behaviour.** By default, Airflow runs all DAG executions that have occurred since the start_date when a DAG is first enabled or re-enabled after being paused. This "catchup" behaviour can flood the executor with historical runs. For most pipelines, disable catchup:

with DAG("my_dag", ..., catchup=False) as dag:

Enable catchup only for DAGs where historical execution is intentional (e.g., migrating from a manual process to Airflow and needing to run the pipeline for historical dates).

**Sensor timeouts.** ExternalTaskSensor and FileSensor (sensors that wait for external conditions) should always have a timeout configured. A sensor without a timeout can wait indefinitely — consuming an executor slot — if the condition it is waiting for never occurs.

Airflow Scheduler and Executor

**Use the CeleryExecutor or KubernetesExecutor for production.** The default LocalExecutor runs all tasks on the scheduler machine and does not scale. CeleryExecutor distributes tasks to worker processes — appropriate for small to medium Airflow deployments. KubernetesExecutor creates a new Kubernetes pod for each task — appropriate for large-scale deployments requiring dynamic resource allocation.

**Monitor the scheduler heartbeat.** The Airflow scheduler logs its heartbeat interval. A slow heartbeat indicates the scheduler is overloaded — either with too many DAGs, too many tasks, or Python code that is slow to parse. Keep DAG count manageable (hundreds, not thousands) and keep DAG file parse time fast (under 1 second per file).

**DAG parse time.** Airflow parses all DAG files continuously. Any code that runs at the module level during import runs on every scheduler heartbeat. Common causes of slow parse time: importing heavy libraries (pandas, spark) at the top of DAG files, database queries during module import, and complex Python logic that runs during DAG construction. Move all execution logic into tasks; keep the DAG file's module-level code to variable declarations and DAG construction.

Monitoring and Alerting

**On-failure callbacks.** Define on_failure_callback at the DAG level for all production DAGs. When any task in the DAG fails, the callback fires — send a Slack message, create a PagerDuty alert, or write to a monitoring table:

def alert_on_failure(context):

dag_id = context["dag"].dag_id

task_id = context["task_instance"].task_id

execution_date = context["execution_date"]

send_slack_alert(f"DAG {dag_id} task {task_id} failed at {execution_date}")

with DAG("critical_pipeline", ..., on_failure_callback=alert_on_failure):

**SLA miss callbacks.** Define sla_miss_callback on DAGs where timeliness is a business requirement. If the DAG does not complete by its SLA, the callback fires even if no individual task has failed:

with DAG("daily_report", ..., sla=timedelta(hours=2)):

This catches pipelines that are running slowly but have not technically failed.

**Airflow metrics with StatsD.** Airflow emits metrics to StatsD: scheduler heartbeat, task success/failure counts, task duration, and queue depth. Route these to Datadog, Prometheus, or Grafana for production monitoring. Alert on scheduler heartbeat gaps (scheduler has stopped) and task queue depth (executor is overloaded).

Common Operational Problems

**Zombie tasks.** A zombie task is a task that the Airflow scheduler believes is running but whose process has died without cleaning up. Zombies are detected and cleaned up by the scheduler's zombie detection mechanism (runs every zombie_detection_interval seconds). If zombie tasks accumulate, check for executor resource exhaustion (workers running out of memory) and ensure task execution timeout is configured.

**DAG dependency cycles.** Circular dependencies between tasks crash the Airflow scheduler. Always test new DAGs locally before deploying. Airflow validates for cycles during DAG parsing and raises an AirflowDagCycleException, which blocks the DAG from loading.

**Metadata database growth.** Airflow stores all DAG runs, task instances, logs, and XCom values in the metadata database. Without periodic cleanup, the database grows large and slows the scheduler. Configure Airflow's built-in metadata cleanup:

airflow db clean --clean-before-timestamp 2024-01-01 00:00:00

Or use the CleanupPlugin for automated periodic cleanup. For high-volume environments, redirect logs to S3 rather than the metadata database.

**XCom overuse.** XCom (cross-communication) allows tasks to pass small values to downstream tasks. It is designed for control signals (pass the file path of the output to the next task) not for data transfer (pass a pandas DataFrame containing millions of rows). XCom values are stored in the metadata database — large XComs bloat the database and slow task communication. If tasks need to share data, use shared storage (S3, warehouse tables) and pass only the reference via XCom.

Our data engineering consulting practice designs and operates production Airflow environments as part of data platform builds — contact us to discuss your pipeline orchestration requirements.

A former Microsoft data architect audits your data foundation, identifies your top priorities, and sends you a written plan. Free. No pitch.