Buildkite: Scalable CI with Your Own Agents

By Kokil Thapa | Last reviewed: September 2026

Buildkite: Scalable CI with Your Own Agents is a hybrid CI model. Buildkite hosts the control plane. Your servers run the build agents. That split matters when you need PHP 8.3 and 8.5 side by side, private MySQL for tests, or Deployer access to production subnets. I've used GitLab CI on sister legal-tech sites and Buildkite-style agent pools on other stacks. The pattern is the same: keep secrets and compute on infrastructure you control. This guide covers architecture, agent install, pipeline YAML, scaling, security, and how Buildkite compares to GitLab CI for Laravel pipelines.

What Is Buildkite and How Does Scalable CI with Your Own Agents Work?

Buildkite is a CI/CD platform founded in 2013. It separates job scheduling from job execution. The SaaS layer stores pipeline definitions, job state, logs, and permissions. Agents on your infrastructure connect outbound to Buildkite and poll for work.

Nothing inbound needs to punch holes through your firewall. Agents initiate TLS connections to Buildkite's API. That outbound-only model is why many teams pick Buildkite over self-hosted Jenkins controllers that expose HTTP ports.

Core components:

• Organization — your Buildkite account and billing boundary.
• Pipeline — a connected repository with steps defined in .buildkite/pipeline.yml or via the UI.
• Agent — a daemon (buildkite-agent) that registers with a token and listens on a queue name.
• Job — one step execution unit assigned to an agent with matching queue tags.
• Build — a single pipeline run triggered by a git push, webhook, API call, or schedule.

When a developer pushes to main, Buildkite creates a build. Each step waits in a queue until an agent with the right tags picks it up. Logs stream back to the Buildkite UI in real time. Failed steps can retry, block downstream work, or trigger notifications.

How Do You Install and Register Buildkite Agents on Your Own Servers?

Agent installation is straightforward on Ubuntu 22.04 or 24.04. You need an agent registration token from Buildkite → Agents → New Agent. Treat that token like a password. Store it in your secrets manager, not in git.

Install the agent on Ubuntu

Official packages live at Buildkite Agent v3 documentation. Typical install flow:

curl -fsSL https://keys.buildkite.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/buildkite-agent-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/buildkite-agent-keyring.gpg] https://apt.buildkite.com/buildkite-agent stable main" | sudo tee /etc/apt/sources.list.d/buildkite-agent.list
sudo apt update && sudo apt install -y buildkite-agent

sudo buildkite-agent bootstrap --token "YOUR_AGENT_TOKEN"

The bootstrap command writes config to /etc/buildkite-agent/buildkite-agent.cfg. Key settings:

token="YOUR_AGENT_TOKEN"
name="prod-ci-01"
tags="queue=laravel,php=8.3,os=linux"
build-path="/var/lib/buildkite-agent/builds"
git-clean-flags="-ffdx"

Start the service:

sudo systemctl enable buildkite-agent
sudo systemctl start buildkite-agent
sudo systemctl status buildkite-agent

Run agents in Docker

Docker agents isolate build environments. Each job gets a clean container. That pattern mirrors what I recommend in self-hosted CI runner security guides.

docker run -d \
  -e BUILDKITE_AGENT_TOKEN="YOUR_AGENT_TOKEN" \
  -e BUILDKITE_AGENT_TAGS="queue=docker,php=8.5" \
  -v /var/run/docker.sock:/var/run/docker.sock \
  buildkite/agent:3

Mount the Docker socket only on dedicated CI hosts. Never share that socket with production app servers. A compromised build script can escape to the host through the socket.

Agent sizing for Laravel projects

For a typical Laravel 12 or 13 app with Pest tests and Vite 8.x builds, plan these baselines per agent:

1. 2 vCPU, 4 GB RAM — lint, unit tests, small Composer installs.
2. 4 vCPU, 8 GB RAM — full test suite with MySQL 8.4 or MariaDB 12.3 service containers.
3. 4 vCPU, 16 GB RAM — parallel Pest runs via ParaTest in CI plus frontend asset builds.

On budget-sensitive Nepal client projects, a single Rs 8,000/month VPS (~USD 60) often runs two lightweight agents. That beats cloud runner minute pricing once you exceed roughly 2,000 build minutes monthly.

How Do You Write a Buildkite Pipeline for Laravel and PHP Projects?

Pipelines are YAML checked into your repository. Buildkite reads .buildkite/pipeline.yml by default. You can also use dynamic pipelines generated by a bootstrap script.

Example pipeline for a Laravel 13 app on PHP 8.3:

env:
  APP_ENV: testing
  DB_CONNECTION: mysql

steps:
  - label: ":composer: Install dependencies"
    command: |
      composer install --no-interaction --prefer-dist --optimize-autoloader
    agents:
      queue: "laravel"
    plugins:
      - docker#v5.15.0:
          image: "php:8.3-cli"
          workdir: /workdir
          mount-checkout: true

  - wait

  - label: ":mysql: Run Pest tests"
    command: |
      cp .env.testing .env
      php artisan migrate --force
      ./vendor/bin/pest --parallel
    agents:
      queue: "laravel"
    artifact_paths:
      - "storage/logs/**/*"
    plugins:
      - docker#v5.15.0:
          image: "php:8.3-cli"
          environment:
            - "DB_HOST=mysql"
          mount-checkout: true

  - label: ":rocket: Deploy with Deployer"
    command: dep deploy production
    agents:
      queue: "deploy"
    branches: "main"
    concurrency: 1
    concurrency_group: "production-deploy"

Notice the agents.queue field. Only agents tagged queue=deploy run the Deployer step. Your app servers never need Buildkite agents. Only a bastion or dedicated deploy runner does.

For database migration safety, follow patterns from database migrations in CI/CD. Run migrations in a dedicated step before traffic-switch deploys. Use concurrency_group to prevent two production deploys at once.

Dynamic pipelines help monorepos. A .buildkite/pipeline.yml can call a script that emits step YAML based on changed paths. That approach aligns with CI/CD for monorepos with multiple PHP apps.

steps:
  - label: ":pipeline: Generate pipeline"
    command: .buildkite/generate-pipeline.sh | buildkite-agent pipeline upload
    agents:
      queue: "laravel"

Buildkite also supports buildkite-agent pipeline upload from any step. The uploaded YAML replaces or extends remaining steps. That is useful when test matrices depend on branch name or changed files.

How Do You Scale Buildkite Agents Without Overpaying for CI?

Scaling Buildkite means adding agents, not rewriting pipelines. Three levers control capacity.

Queue-based routing

Tags route jobs to the right hardware. Common tag scheme:

• queue=default — general Linux builds.
• queue=laravel — PHP-FPM tuned images with Composer cache.
• queue=deploy — SSH keys for Deployer 7, restricted sudo.
• queue=heavy — 8+ vCPU machines for ParaTest or Magento 2.4.x compiles.

Steps declare agents: { queue: "laravel" }. Buildkite never sends a deploy job to a test-only agent.

Horizontal scaling with auto-scaling groups

On AWS or similar clouds, put agents in an Auto Scaling Group. Scale on queue depth metrics. Buildkite exposes API endpoints and webhooks for queue monitoring. A Lambda or cron job can call aws autoscaling set-desired-capacity when pending jobs exceed a threshold.

I've seen this pattern on production Laravel applications where nightly builds spike but daytime load is low. You pay for eight agents during the deploy window and two agents overnight.

Elastic agents via Docker and spot instances

Spot or preemptible VMs cut cost 60–70%. Wrap them with a startup script that registers an agent on boot and deregisters on shutdown. Pair spot agents with retry-friendly steps. Buildkite retries failed jobs on healthy agents automatically when you enable step retries.

For teams already running Jenkins distributed builds with agents, the mental model transfers directly. Buildkite queues replace Jenkins labels. The difference is you do not maintain a Java controller VM.

How Does Buildkite Compare to GitLab CI, Jenkins, and GitHub Actions?

Pick Buildkite when you want managed orchestration plus full control over execution hardware. Pick all-in-one platforms when you want zero agent maintenance.

GitLab CI remains my default for sister sites on a shared Deployer 7 pipeline. Those repos already live in GitLab. Buildkite earns its place when the git host is GitHub or Bitbucket and the team still wants private-network test databases. Read the full breakdown in GitHub Actions vs GitLab CI comparison and Bitbucket Pipelines CI/CD guide.

Buildkite pricing charges per active user, not per build minute on your agents. Heavy test suites on your own hardware can cost less than cloud minutes at scale. You still pay for the VMs themselves. Use a JSON formatter to inspect Buildkite API responses when scripting auto-scaling.

How Do You Secure Buildkite Agents and Manage Secrets in Production CI?

Self-hosted agents execute arbitrary code from your repository. Compromised code equals compromised agent. Treat CI hosts as a dedicated security zone.

Agent hardening checklist

1. Run agents on dedicated CI machines, not production app servers.
2. Use separate agent tokens per environment (staging vs production deploy).
3. Mount SSH deploy keys with read-only git access where possible.
4. Enable git-clean-flags="-ffdx" so every build starts from a clean checkout.
5. Restrict queue=deploy agents to a single bastion with UFW allowing only SSH outbound.
6. Rotate agent tokens quarterly and after any team member departure.

Secrets belong in Buildkite's encrypted environment variables or in a vault your pipeline reads at runtime. Never commit .env files. Follow secrets handling in CI/CD pipelines and CI/CD secrets management best practices.

steps:
  - label: ":lock: Deploy"
    env:
      DEPLOY_KEY:
        from_secret: production-deploy-key
    command: dep deploy production
    agents:
      queue: "deploy"

Add Gitleaks secrets scanning as an early pipeline step. Block merges when credentials appear in diffs. Pair that with DevSecOps shift-left practices for dependency auditing via composer audit.

For coverage enforcement, gate merges with code coverage gates in CI. Buildkite's block step pauses the pipeline until a human approves risky deploys. That manual gate works well for legal-tech portals where a bad deploy affects live client document uploads.

If you need hands-on server hardening for CI hosts, Linux system administration covers UFW, fail2ban, and PHP-FPM tuning on Ubuntu. For full pipeline design on client projects, see enterprise application development.

What Buildkite Patterns Work on Real Laravel and eCommerce Projects?

On a production Laravel application, I group pipelines into three tiers. Pull requests run lint and unit tests only. Merges to develop run integration tests against a disposable MySQL schema. Merges to main deploy through Deployer after a manual block step.

For WooCommerce 11.1 or Magento 2.4.x shops, dedicate a queue=heavy agent with more RAM. Asset compilation and static content deploy steps consume memory fast. The Quick And Easy Nepalese Grocery Laravel eCommerce project and Adventure Third Pole Trek booking platform both benefit from split test and deploy queues even on GitLab CI. The same queue pattern applies on Buildkite.

Blue-green deploys pair naturally with Buildkite. Run health checks before switching traffic. Details sit in CI/CD blue-green deployment explained. For Pest-specific config, see Laravel testing with Pest in CI/CD.

Teams evaluating Azure-hosted agents should also read self-hosted Azure DevOps agents. The networking and token rotation patterns overlap. Buildkite's plugin ecosystem covers Docker, AWS ECR, Slack notifications, and artifact uploads to S3. Official plugin docs live at Buildkite pipeline plugins documentation.

When a client outgrows a single VPS, Buildkite avoids vendor lock-in on compute. You keep the same pipeline YAML while moving agents from a Kathmandu data centre to AWS ap-south-1. That portability matters for Nepal businesses balancing local latency against global redundancy.

Ongoing agent updates, token rotation, and pipeline tuning fall under support and maintenance. Document every queue tag and agent hostname in your runbook. Future you will thank present you at 2 a.m. when a deploy queue drains to zero agents.

Key Takeaways

• Buildkite splits CI orchestration (SaaS) from execution (your agents), giving scalable CI without inbound firewall rules.
• Install buildkite-agent on dedicated Linux hosts or Docker runners; route jobs with queue tags.
• Store pipelines in .buildkite/pipeline.yml; use separate queues for tests, heavy builds, and Deployer production deploys.
• Scale horizontally with auto-scaling groups; use spot instances for tests and on-demand VMs for deploy agents.
• Isolate deploy credentials, rotate agent tokens, and scan for secrets before any production step runs.
• Buildkite fits GitHub or Bitbucket teams that outgrow cloud runner minute pricing but do not want Jenkins controller maintenance.

People Also Ask

Is Buildkite free to use?

Buildkite offers a trial period for new organizations. Production use requires a paid plan based on active users, not agent count or build minutes on your hardware. You pay separately for the servers running your agents. Small teams on one VPS often spend less total than all-cloud CI at high build volume.

Can Buildkite agents run inside Kubernetes?

Yes. You can run agents as Kubernetes pods or use the Buildkite agent stack Helm chart. Each pod registers with a queue tag. Jobs spawn as sibling pods via the Kubernetes plugin. This pattern suits teams already running Tekton Kubernetes-native CI/CD who want Buildkite's UI and permissions model instead.

Does Buildkite work with GitHub and Bitbucket?

Buildkite integrates with GitHub, Bitbucket, GitLab, and plain Git webhooks. Connect the repository in Buildkite settings. Pushes trigger builds automatically. Branch filters and path filters in pipeline YAML limit which events start expensive test suites.

How is Buildkite different from Jenkins?

Jenkins requires you to host and patch the controller application. Buildkite hosts the controller as SaaS. Both use self-hosted agents for execution. Buildkite agents connect outbound only. Jenkins often exposes a web UI port that needs VPN or IP restriction. Buildkite trades Jenkins plugin breadth for lower operational overhead on the control plane.

Ship Scalable CI on Infrastructure You Control

Buildkite: Scalable CI with Your Own Agents gives you a managed scheduler and full freedom over execution hardware. Install agents on Ubuntu, tag your queues, write pipeline.yml, and scale by adding machines—not by rewriting your entire CI platform. Start with a single test agent and a separate deploy agent. Add spot-backed pools once build volume grows.

If you want help designing Buildkite pipelines for Laravel 13, hardening agent hosts, or migrating from Jenkins or GitLab CI, contact us to talk through your stack. You can also browse the Notary Kathmandu portfolio for examples of production Deployer workflows on shared infrastructure.