Airflow Dag Patterns

Apache Airflow DAG Patterns

Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.

Use this skill when

• Creating data pipeline orchestration with Airflow
• Designing DAG structures and dependencies
• Implementing custom operators and sensors
• Testing Airflow DAGs locally
• Setting up Airflow in production
• Debugging failed DAG runs

Do not use this skill when

• You only need a simple cron job or shell script
• Airflow is not part of the tooling stack
• The task is unrelated to workflow orchestration

Instructions

• Identify data sources, schedules, and dependencies.
• Design idempotent tasks with clear ownership and retries.
• Implement DAGs with observability and alerting hooks.
• Validate in staging and document operational runbooks.

Refer to resources/implementation-playbook.md for detailed patterns, checklists, and templates.

Safety

• Avoid changing production DAG schedules without approval.
• Test backfills and retries carefully to prevent data duplication.

Resources

• resources/implementation-playbook.md for detailed patterns, checklists, and templates.

Limitations

• Use this skill only when the task clearly matches the scope described above.
• Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
• Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.