Apache Airflow: Orchestrate Data Pipelines

By Kokil Thapa | Last reviewed: September 2026

Your nightly ETL job failed at 2 a.m. again. The warehouse sync ran, but the report never refreshed because nobody wired the steps together. Apache Airflow: Orchestrate Data Pipelines solves that problem by treating workflows as code. You define a directed acyclic graph (DAG) in Python, schedule it, and let Airflow handle retries, dependencies, and observability. If you already run GitLab CI pipelines for Laravel or batch exports from a production app, Airflow sits one layer above: it coordinates long-running data work across databases, APIs, and cloud storage—not just build and deploy steps.

What Is Apache Airflow and How Does It Orchestrate Data Pipelines?

Apache Airflow is an open-source workflow orchestrator originally built at Airbnb and now maintained under the Apache Software Foundation. It does not move data itself. Instead, it decides when each step runs, what happens if a step fails, and whether downstream tasks should proceed.

Think of it as a production cron replacement with a UI, audit trail, and dependency graph. A typical pipeline might extract rows from MySQL, transform them in Python, load them into a warehouse, then trigger a dbt run or send a Slack alert. Each step is a task. Edges between tasks encode order.

On client projects where I maintain Laravel applications with heavy reporting, I often see teams start with ad-hoc shell scripts and cron. That works until dependencies multiply. Airflow becomes worthwhile when you have five or more recurring jobs that share data, overlap in time, or need coordinated failure handling. For simple “run this script nightly” jobs, Laravel’s scheduler or a single CI cron may still be enough.

Core components you must understand

• DAG — a Python file describing tasks and dependencies. The graph must be acyclic; loops require careful design.
• Task / Operator — a unit of work. BashOperator, PythonOperator, and provider packages like PostgresOperator wrap common actions.
• Scheduler — parses DAGs, creates DAG runs, and queues ready tasks based on schedule and upstream success.
• Executor — decides where tasks run: locally for dev, Celery or Kubernetes for production scale.
• Metadata database — stores DAG definitions, run state, variables, connections, and XCom payloads.

The official project documentation at airflow.apache.org remains the authoritative reference for operators, configuration keys, and upgrade paths. Airflow 2.x replaced the experimental API with a stable REST interface and improved the UI significantly; most greenfield projects in 2026 should start on the current 2.x line rather than legacy 1.10 installs.

How Do You Write Your First Airflow DAG for a Data Pipeline?

Start local. Docker Compose is the fastest path for learning without polluting your laptop Python environment. Install Docker, clone the official compose template, and bring up webserver, scheduler, and Postgres metadata in one command.

curl -LfO https://airflow.apache.org/docs/apache-airflow/2.10.3/docker-compose.yaml
mkdir -p ./dags ./logs ./plugins
echo -e "AIRFLOW_UID=$(id -u)" > .env
docker compose up airflow-init
docker compose up -d

Create dags/daily_sales_etl.py with a minimal three-step pipeline: extract, transform, load. Use the TaskFlow API (@task decorators) for readable Python-native DAGs instead of verbose operator boilerplate.

from airflow.decorators import dag, task
from pendulum import datetime

@dag(
    start_date=datetime(2026, 1, 1),
    schedule="0 2 * * *",
    catchup=False,
    tags=["sales", "etl"],
)
def daily_sales_etl():
    @task
    def extract():
        return [{"sku": "A1", "qty": 3}, {"sku": "B2", "qty": 7}]

    @task
    def transform(rows: list):
        return [{"sku": r["sku"], "total": r["qty"] * 10} for r in rows]

    @task
    def load(rows: list):
        for row in rows:
            print(f"LOAD {row}")

    load(transform(extract()))

daily_sales_etl()

Open the web UI at port 8080. Toggle the DAG on. Trigger a manual run. Confirm task boxes turn green in order. That green path is your contract: upstream success unlocks downstream work.

Production-minded DAG habits

1. Set catchup=False unless you truly need historical backfill; accidental catchup can hammer APIs.
2. Use default_args with retries, retry_delay, and owner on every DAG.
3. Store secrets in Airflow Connections or an external vault—not in DAG source files.
4. Keep DAG files idempotent; reruns should not double-load rows without dedup keys.
5. Pin provider package versions in requirements.txt alongside your Airflow version.

For JSON payloads moving between tasks, validate structure early. A quick pass through a JSON formatter and validator during development catches schema drift before it hits production logs.

When Should You Choose Airflow Over Laravel Queues or Step Functions?

Not every project needs Airflow. The decision depends on workflow complexity, team skills, and runtime environment. Laravel queues excel at application-side jobs: send email, resize image, sync one order. Airflow excels at cross-system batch orchestration with calendars, SLAs, and operational visibility.

If your data pipeline lives inside a Laravel 12 or 13 app and touches one database, start with queues. When jobs span Postgres, S3, Snowflake, and a third-party API with strict ordering, Airflow earns its keep. For AWS-only event chains without Python ops capacity, compare with orchestrating workflows with AWS Step Functions.

How Do You Deploy Airflow to Production in 2026?

Local Docker Compose teaches concepts. Production needs hardened executors, isolated secrets, monitored workers, and version-controlled DAG deployment separate from ad-hoc server edits.

Executor choice

LocalExecutor runs tasks as subprocesses on one machine. Fine for small teams with modest volume. CeleryExecutor distributes tasks across a worker pool backed by Redis or RabbitMQ— a pattern familiar if you already run Redis for caching and data structures in Laravel apps. KubernetesExecutor spins a pod per task; ideal for spiky ML workloads and teams running ML pipelines on Kubernetes.

Configuration essentials

# airflow.cfg excerpts — store in env-specific templates
[core]
executor = CeleryExecutor
load_examples = False
dags_are_paused_at_creation = True

[scheduler]
min_file_process_interval = 30
dag_dir_list_interval = 60

[webserver]
expose_config = False
warn_deployment_exposure = True

Store the metadata database on PostgreSQL 18 or MySQL 9.7 with regular backups. Never use SQLite outside single-user dev. Mount DAGs from Git via CI/CD—similar discipline to build pipeline automation best practices you would apply for application deploys.

Managed options like Astronomer, AWS MWAA, and Google Cloud Composer reduce ops burden. Budget roughly USD 300–800/month (~Rs 40,000–105,000) for small managed clusters versus self-hosting on a USD 80 VPS if you accept on-call responsibility. Nepali startups often self-host on a single Ubuntu 24 box until pipeline failure cost exceeds engineer time.

What Are Common Airflow Mistakes That Break Data Pipelines?

Airflow is powerful and easy to misuse. These failures show up repeatedly in production postmortems.

Putting business logic inside the DAG file

DAG files should wire tasks, not implement 400 lines of transformation. Import reusable Python modules. Test those modules with pytest outside Airflow. Keep the DAG thin so parsing stays fast.

Treating XCom as a data bus

XCom stores small metadata between tasks— not megabyte CSV blobs. Pass S3 paths or database keys instead. Large XCom payloads slow the metadata database and break workers.

Ignoring data intervals and backfill

Airflow 2.x uses data intervals for logical run windows. Misunderstanding execution_date versus data_interval_start causes off-by-one-day bugs in reports. Document timezone handling explicitly; Nepal teams exporting to UTC warehouses need consistent conversion, similar to challenges when syncing Bikram Sambat dates through a Nepali date converter in app layers.

Skipping idempotency and data quality checks

Add a ShortCircuitOperator or validation task before load steps. Integrate expectations from Great Expectations or simple row-count assertions. Pair this with test data management for pipelines so staging runs mirror production shape without leaking PII.

Running Airflow when simpler tools suffice

A single nightly mysqldump plus rsync does not need a DAG. Over-engineering ops for two tasks creates failure surface without benefit. Match tool to problem size.

How Do You Connect Airflow to Laravel and Web Application Data?

Most web teams already have a Laravel or WordPress application as the system of record. Airflow sits beside it, not inside it.

Typical integration patterns:

• Database extract — Airflow PostgresOperator or MySqlOperator reads replica tables Laravel writes. Use read replicas to avoid load on primary MySQL 9.7 instances.
• HTTP extract — call a secured Laravel API endpoint built with REST API development practices and paginate results into staging tables.
• File drop — Laravel exports CSV to S3 nightly; Airflow S3KeySensor waits for the object, then loads the warehouse. This decouples release cycles.
• Trigger from app — Laravel dispatches a message, but Airflow owns the heavy multi-step pipeline. Avoid duplicating orchestration in both places.

On an eCommerce project like Quick And Easy Nepalese Grocery, order and inventory data naturally lives in the application database. Airflow would aggregate that data for finance and analytics without slowing checkout queries. The app keeps serving users; the pipeline handles batch truth.

For embedding or AI enrichment workflows, chain Airflow upstream of vector index builds described in building an embeddings pipeline. Batch document fetch and chunking fit Airflow; real-time inference belongs in API workers.

If you need help designing the boundary between app code and batch orchestration, custom software development that treats data flow as architecture—not an afterthought— saves months of rework.

Key Takeaways

• Apache Airflow orchestrates data pipelines by scheduling Python DAGs with explicit task dependencies, retries, and a full run history UI.
• Start with Docker Compose locally, use the TaskFlow API, and enforce idempotent tasks before production deployment.
• Choose Airflow over Laravel queues when pipelines span multiple systems, need SLAs, or require operational visibility beyond application logs.
• Deploy with CeleryExecutor or KubernetesExecutor, PostgreSQL metadata, Git-synced DAGs, and secrets stored outside source control.
• Never pass large datasets through XCom; use object storage paths and validate row counts before warehouse loads.
• Integrate Airflow beside your web app via replicas, file drops, or secured APIs—not by stuffing business logic into DAG files.

People Also Ask

Is Apache Airflow free to use?

Yes. Airflow is open source under the Apache 2.0 license. You pay for infrastructure, engineer time, or a managed service such as AWS MWAA or Astronomer. Self-hosting on a VPS can cost under USD 100/month (~Rs 13,000), while managed tiers add convenience and support at higher monthly fees.

Does Airflow require Python?

DAG definitions are Python files, and most operators assume a Python runtime. You can still invoke Bash, SQL, or Docker operators to run non-Python tools. Teams standardized on PHP for application code often keep a small Python repo dedicated to pipeline DAGs rather than forcing Airflow into a Laravel codebase.

What is the difference between Airflow and Prefect or Dagster?

All three orchestrate workflows as code. Airflow has the largest operator ecosystem and self-hosted maturity. Prefect emphasises a modern Python API with dynamic flows. Dagster treats data assets and lineage as first-class concepts. Airflow wins when you need broad integrations and a battle-tested scheduler; alternatives win when developer ergonomics or asset-centric modelling is the priority.

Can Airflow replace Jenkins or GitLab CI?

No. CI/CD tools build, test, and deploy application artifacts on commit. Airflow schedules recurring data workflows on timetables or dataset triggers. Use Jenkins or Bitbucket Pipelines for deploy pipelines, and Airflow for ETL and analytics jobs that run on business calendars—not every git push.

Build Pipelines That Survive 2 a.m.

Batch failures are rarely mysterious SQL bugs. They are missing dependencies, silent cron overlap, and scripts nobody owns. Apache Airflow: Orchestrate Data Pipelines gives your team a single place to define order, retries, and accountability. Start small—one DAG, three tasks, real data from a read replica. Harden from there with CI-synced DAGs, monitored workers, and clear boundaries between app code and orchestration.

If you are wiring Laravel apps, warehouse loads, and third-party APIs into a coherent pipeline, I can help architect and deploy that stack. See enterprise application development and AI integration and automation services, browse the portfolio for shipped systems, or contact us to discuss your data workflow requirements.