Schedule Pipelines with Dagster

By Kokil Thapa | Last reviewed: September 2026

You need to schedule pipelines with Dagster when batch jobs must run on a clock, not on a developer laptop. Raw Linux cron works for one script. It breaks down once you need run history, dependency graphs, retries, and safe backfills. Dagster treats schedules as first-class code assets tied to jobs and ops. This guide walks through defining cron schedules, running the daemon, and choosing sensors when events beat the clock. If you already orchestrate with Apache Airflow for data pipelines, the mental model is familiar—but Dagster keeps schedules beside the job definitions you already version in Git.

What do you need before you schedule pipelines with Dagster?

Dagster is a Python orchestrator. Your pipeline code lives in a Dagster project—typically a definitions.py or module the workspace loads. You need Python 3.9+ (3.11 or 3.12 is common on new projects in 2026), Dagster installed, and a job you want to run on a timer.

On a real client project, I treat scheduling like any production cron replacement: version the schedule in Git, run it through the same deploy path as the app, and log every tick. That mirrors how I wire Linux cron jobs for Laravel queues—but Dagster adds a UI, run storage, and lineage.

Minimal project layout

my_dagster_project/
├── pyproject.toml
├── my_dagster_project/
│   ├── __init__.py
│   ├── assets.py          # @asset definitions
│   ├── jobs.py            # define_asset_job(...)
│   ├── schedules.py       # ScheduleDefinition / @schedule
│   └── definitions.py     # Definitions(..., schedules=[...])
└── workspace.yaml

Install Dagster with pip or uv:

pip install dagster dagster-webserver dagster-graphql

Local dev uses two processes: the web UI and the daemon that evaluates schedules and sensors.

1. Create a job from assets or ops.
2. Attach a schedule with a cron string or partition logic.
3. Register both in Definitions.
4. Start dagster-daemon run alongside the webserver.
5. Enable the schedule (stopped by default in many setups).

For JSON config payloads passed between ops, validate shapes early with a JSON formatter during development. Bad config at 2 AM is worse than a failed unit test at 2 PM.

How do you define a cron schedule for a Dagster job?

Dagster accepts standard five-field cron strings (minute, hour, day of month, month, day of week). Use UTC in production unless you document timezone overrides clearly. Nepal teams often schedule around NPT (UTC+5:45)—convert carefully or use Dagster's timezone-aware scheduling options in your deployment config.

ScheduleDefinition example

# schedules.py
from dagster import ScheduleDefinition, DefaultScheduleStatus, define_asset_job
from .assets import daily_sales_assets

daily_sales_job = define_asset_job(
    name="daily_sales_job",
    selection=[daily_sales_assets],
)

daily_sales_schedule = ScheduleDefinition(
    job=daily_sales_job,
    cron_schedule="0 2 * * *",  # 02:00 UTC daily
    name="daily_sales_schedule",
    default_status=DefaultScheduleStatus.STOPPED,
    tags={"team": "analytics", "env": "prod"},
)

Decorator style with run config

from dagster import schedule, RunRequest, SkipReason
import datetime

@schedule(cron_schedule="30 6 * * 1-5", job=daily_sales_job)
def weekday_morning_schedule(context):
    # Skip Nepal public holidays manually or via external calendar
    if context.scheduled_execution_time.weekday() >= 5:
        return SkipReason("Weekend")
    return RunRequest(
        run_key=f"weekday-{context.scheduled_execution_time:%Y-%m-%d}",
        tags={"scheduled": "true"},
        run_config={
            "ops": {
                "load_raw_sales": {
                    "config": {"source_date": context.scheduled_execution_time.strftime("%Y-%m-%d")}
                }
            }
        },
    )

Register schedules in your entry point:

# definitions.py
from dagster import Definitions
from .jobs import daily_sales_job
from .schedules import daily_sales_schedule, weekday_morning_schedule
from .assets import daily_sales_assets

defs = Definitions(
    assets=[daily_sales_assets],
    jobs=[daily_sales_job],
    schedules=[daily_sales_schedule, weekday_morning_schedule],
)

Point workspace.yaml at that module:

load_from:
  - python_module: my_dagster_project.definitions

Start locally:

dagster dev -m my_dagster_project.definitions

The dev command runs webserver and daemon together. That convenience hides a production requirement: the daemon is a separate long-lived service. I've seen the same class of bug on GitLab CI deploy pipelines—the app ships, but the worker process never restarts.

Schedules vs sensors vs manual launches

Official reference: Dagster's schedule docs at docs.dagster.io/concepts/partitions-schedules-sensors/schedules.

How does the Dagster daemon execute scheduled pipeline runs?

The daemon is the heartbeat. It reads schedule definitions, compares cron ticks to the last launch time, and submits runs to your run launcher. Without it, schedules appear in the UI but never fire. This is the single most common production misconfiguration.

Production daemon setup

# systemd unit example — /etc/systemd/system/dagster-daemon.service
[Unit]
Description=Dagster Daemon
After=network.target

[Service]
User=dagster
WorkingDirectory=/opt/dagster/app
Environment=DAGSTER_HOME=/opt/dagster/dagster_home
ExecStart=/opt/dagster/venv/bin/dagster-daemon run
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Set DAGSTER_HOME to a persistent directory holding dagster.yaml and storage config. On Ubuntu servers I maintain for clients, I follow the same discipline as Linux system administration work: dedicated service user, log rotation, and health checks.

Configure run storage and compute in dagster.yaml:

# $DAGSTER_HOME/dagster.yaml
storage:
  postgres:
    postgres_db:
      username: dagster
      password: {env: DAGSTER_PG_PASSWORD}
      hostname: localhost
      db_name: dagster
      port: 5432

run_launcher:
  module: dagster.core.launcher
  class: DefaultRunLauncher

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 5

PostgreSQL backs run history for teams outgrowing SQLite. The pattern parallels test data management for pipelines—isolate environments and never point staging schedules at production databases.

Daemon reference: docs.dagster.io/deployment/dagster-daemon.

When should you use partitioned schedules instead of a simple cron?

Simple cron runs one job instance per tick. Partitioned schedules run one slice per tick—ideal for daily date partitions, per-region loads, or incremental backfills.

Daily partitions example

from dagster import DailyPartitionsDefinition, build_schedule_from_partitioned_job

daily_partitions = DailyPartitionsDefinition(start_date="2025-01-01")

partitioned_job = define_asset_job(
    name="partitioned_etl_job",
    selection=[daily_sales_assets],
    partitions_def=daily_partitions,
)

daily_partition_schedule = build_schedule_from_partitioned_job(
    job=partitioned_job,
    hour_of_day=3,
    minute_of_hour=0,
)

Each tick materializes the partition matching the schedule date. You can backfill a date range from the UI without writing one-off scripts. That saves hours when reconciling late-arriving data.

For ML feature stores or embedding indexes, partitioned schedules pair well with patterns from building an embeddings pipeline. Each day gets its own reproducible run key.

How do sensors complement scheduled Dagster pipelines?

Schedules answer "run at 3 AM." Sensors answer "run when something changed." A sensor polls an external system—S3 prefix, Kafka topic, or upstream table—and yields RunRequest objects when criteria match.

from dagster import sensor, RunRequest, SkipReason
import boto3

@sensor(job=daily_sales_job, minimum_interval_seconds=300)
def s3_file_sensor(context):
    s3 = boto3.client("s3")
    prefix = "landing/sales/"
    resp = s3.list_objects_v2(Bucket="analytics-raw", Prefix=prefix, MaxKeys=1)
    if resp.get("KeyCount", 0) == 0:
        return SkipReason("No new files")
    return RunRequest(run_key=f"s3-{context.cursor or 'init'}")

Sensors also need the daemon. Set minimum_interval_seconds to avoid hammering APIs. Store cursor state in the sensor context so you do not double-process files.

Combine both: a nightly partition schedule for baseline loads, plus a sensor for intraday refreshes when vendors drop files early. That hybrid shows up in analytics platforms and in AI integration and automation flows where embeddings refresh after document uploads.

What production practices keep scheduled Dagster pipelines reliable?

Scheduling is operations, not just Python. Treat missed ticks, stale secrets, and run queue saturation as normal failure modes.

Secrets and config

Never hard-code database passwords in schedule run config. Use environment variables resolved at launch, or a secrets backend your platform supports. The same rules apply as when you manage secrets safely in pipelines or handle secrets in CI/CD.

Idempotency and run keys

Pass a stable run_key from schedules so Dagster deduplicates accidental double launches. Design ops to tolerate retries—upserts beat blind inserts. This mirrors idempotent deploy steps in build pipeline automation best practices.

Monitoring checklist

• Daemon process alive (systemd or k8s liveness probe).
• Schedule status RUNNING in UI—not STOPPED after redeploy.
• Alert on run failures via Slack/email integration or Dagster Cloud alerts.
• Track duration trends; sudden spikes often mean bad upstream data.
• Keep max_concurrent_runs below database connection limits.

For Kubernetes deployments, run coordinators and launchers resemble patterns in Kubeflow ML pipelines on Kubernetes—one control plane, many workers. Dagster Cloud removes daemon hosting if your team prefers managed ops (~USD 100–500/month depending on scale; budget Rs 13,000–65,000/month for Nepal teams comparing build-vs-buy).

Database migration steps should never run inside unreviewed schedule code without the safeguards you'd use for database migrations in CI/CD pipelines. Promote schedule changes through dev → staging → prod like any other code path described in infrastructure promotion pipelines.

On booking and reporting systems I've shipped—such as Adventure Third Pole Trek—nightly aggregation jobs must survive deploys and holidays. Dagster schedules give operators a visible switch and a replay button. That beats grep-ing cron on three servers.

If your team needs a custom orchestration layer around web apps and data jobs, see custom software development or ongoing support and maintenance. For broader CI context, compare GitHub Actions vs Azure Pipelines and Argo Workflows for CI pipelines.

Key Takeaways

• Define schedules in code beside jobs; register them in Definitions and load via workspace.yaml.
• Run dagster-daemon run as a persistent service—schedules never fire without it.
• Use partitioned schedules for date-sliced ETL and one-click backfills from the UI.
• Add sensors when files or events should trigger runs between cron ticks.
• Set stable run_key values, externalize secrets, and alert on failed runs.
• Enable schedules explicitly after deploy; default status is often STOPPED.

People Also Ask

Does Dagster use cron syntax for schedules?

Yes. Dagster schedules accept standard five-field cron strings. You can also use the @schedule decorator to return RunRequest, SkipReason, or multiple requests per tick for advanced control.

What is the difference between Dagster schedules and sensors?

Schedules are time-driven—they fire on cron ticks evaluated by the daemon. Sensors are event-driven—they poll external systems or watch asset materializations and launch runs when conditions match.

Can Dagster schedules run in Docker or Kubernetes?

Yes. The daemon submits runs to the configured run launcher. Docker and Kubernetes launchers spin isolated containers or pods per run while the daemon stays lightweight on a control node.

How do you backfill missed days in Dagster?

Use partitioned schedules and select a date range in the Dagster UI to launch historical partitions. For non-partitioned jobs, trigger manual runs with explicit run config for each date you need to reprocess.

Ship reliable scheduled pipelines

You schedule pipelines with Dagster by codifying cron triggers, keeping the daemon alive, and choosing partitions or sensors based on how your data arrives—not how demos look in tutorials. Start with one job, one schedule, and PostgreSQL run storage before you add ten sensors and wonder why the queue backs up.

Need help designing data jobs beside your Laravel or eCommerce platform—or hardening production orchestration? Contact us to talk through architecture, or browse the blog for more pipeline guides. You can also read about the author on About Me and review shipped work in the portfolio.