September 11, 2026
12 min read
By Kokil Thapa | Last reviewed: September 2026
Production breaks at the worst moment. A payment callback stalls, PHP-FPM exhausts workers, or a deploy leaves cron jobs pointing at an old release path. When that happens, you need more than tribal knowledge. You need to write effective runbooks that turn panic into a short checklist. On real client projects I maintain with support and maintenance contracts, runbooks are the difference between a ten-minute fix and a two-hour outage. This guide shows how to structure, test, and keep runbooks that actually work when someone reads them at 2 a.m.
Quick answer: To write effective runbooks, document one failure mode per page with clear triggers, severity, copy-paste commands, expected output, rollback steps, and owner contacts. Test them during calm periods and after every deploy change.
How Do You Write Effective Runbooks for Web Applications?
A runbook is an operational playbook. It tells an engineer what to do when a specific alert fires or a user report matches a known pattern. It is not architecture documentation. It is not a wiki dump. It is a sequence of verified steps that reduce mean time to recovery.
Start with the incidents you already hit twice. Queue backlog growth, disk full on /var, SSL renewal failure, stale opcache after deploy, or a webhook retry storm. Each deserves its own runbook file. One symptom, one owner service, one resolution path.
I keep runbooks in the same Git repo as infrastructure notes or in a dedicated docs/runbooks/ folder. Markdown works well. You can render it in GitLab, Confluence, or a static site. Pair each runbook with the alert that should open it. If the alert and the doc live in different systems, link them explicitly in both directions.
Pick the right scope
Good runbooks answer one question: "What do I do when X happens?" Bad runbooks try to cover everything about Laravel, MySQL, and Nginx on one page. Split by failure mode instead.
- Symptom-led: "502 Bad Gateway on checkout" — starts from what the user or monitor sees.
- Alert-led: "DiskUsageCritical on prod-web-01" — starts from the Prometheus or Zabbix rule name.
- Procedure-led: "Rotate database credentials" — starts from a scheduled task, not an incident.
For Linux system administration work, I usually mix symptom-led and alert-led formats. The on-call engineer may arrive from either path.
Use a consistent template
Every runbook in your library should share the same headings. Predictable structure beats clever prose at 2 a.m. Here is a template I reuse on production Laravel stacks:
# RB-042: Laravel queue workers stopped processing
## Metadata
- Severity: P2 (orders delayed, site up)
- Service: app.example.com / queue:default
- Owner: @platform-team
- Last tested: 2026-08-14
- Related alert: QueueDepthHigh
## Trigger
Horizon shows zero active workers OR queue depth > 500 for 10 minutes.
## Impact
Background jobs stall: emails, PDFs, payment reconciliation.
## Prerequisites
SSH access to prod-app-01, sudo for systemctl.
## Diagnosis
1. ssh deploy@app-server
2. sudo systemctl status laravel-worker@default
3. tail -n 100 /var/www/app/shared/storage/logs/worker.log
## Resolution
1. sudo systemctl restart laravel-worker@default
2. php artisan queue:restart
3. Confirm Horizon dashboard shows active workers
## Verification
- Queue depth drops below 50 within 5 minutes
- Test job: php artisan tinker → dispatch a ping job
## Rollback
If restart fails, scale to backup worker node per RB-019.
## Escalation
PagerDuty: Platform Primary → Secondary after 15 min
Store templates in Git. Review them like code. A broken command in a runbook is a production bug waiting to happen.
What Should Every Effective Runbook Include?
Missing sections cause hesitation. Hesitation extends outages. After reviewing runbooks across sister sites on a shared Deployer 7 pipeline — including Notary Kathmandu and similar legal-tech portals — these fields show up in every doc that actually gets used.
Metadata and ownership
Put severity, service name, owner team, and last-tested date at the top. On-call staff need to know whether they can fix it alone or must wake a DBA. Last-tested date tells them whether the steps still match production.
Copy-paste commands with expected output
Never write "check the logs." Write the exact path and the line that confirms health. Example:
grep "Ready to handle connections" /var/log/php8.4-fpm.log | tail -1
# Expected: [11-Sep-2026 08:00:01] NOTICE: ready to handle connections
Include hostnames, usernames, and PHP binary paths. Production servers often run PHP 8.4 while your laptop runs 8.5. Wrong binary, wrong result.
Rollback and escalation
Every mutating step needs a rollback note. Restarting PHP-FPM is low risk. Truncating a table is not. State the blast radius plainly. Then list who to call and after how many minutes. The Google SRE incident response chapter treats escalation paths as first-class incident tooling. Your runbook should too.
How Do You Structure Runbooks for On-Call Engineers?
On-call readers are tired, possibly on mobile, and under time pressure. Structure for scanability, not literary flow. Put the fastest safe fix near the top if you use a tiered format.
Three-tier layout
- Tier 1 — Quick fix: One to three steps that fix 80% of occurrences. Example: restart PHP-FPM after deploy.
- Tier 2 — Deeper diagnosis: Log paths, SQL queries,
curlhealth checks, Redis ping. - Tier 3 — Escalation: When to stop, what data to collect, who owns the root cause.
This mirrors the approach in the PagerDuty on-call guide. Front-load action. Bury theory.
Link related runbooks and tools
Incidents chain. A deploy runbook should link to opcache invalidation and queue restart docs. Payment failures should link to webhook replay procedures. Internal cross-links beat searching a wiki.
For JSON log parsing during diagnosis, I point engineers to a JSON formatter tool so they can pretty-print API responses without leaving the browser. Small tooling links save minutes during incidents.
See also the companion post on on-call and incident response runbooks for paging policy and comms templates.
| Runbook type | Best for | Update trigger | Typical owner |
|---|---|---|---|
| Incident response | Alerts, outages, degraded service | After every related incident | Platform / DevOps |
| Deployment / rollback | Release failures, bad builds | Every pipeline change | Release manager |
| Maintenance procedure | Cert renewal, DB failover drills | Quarterly review | SRE or sysadmin |
| Security response | Credential leak, suspected breach | After audit or pen test | Security + platform |
How Do You Write Runbooks for Laravel and PHP Production Stacks?
Most of my runbooks target Laravel 12 or 13 on Ubuntu with Apache or Nginx and PHP-FPM 8.3+. Stack-specific paths belong in the doc. Generic advice fails when storage/ permissions or opcache stale code is the root cause.
Deploy and symlink failures
Deployer 7 uses symlinked releases. Cron jobs that hard-code an old release path are a pattern I have seen repeatedly. Document the fix:
ls -la /var/www/example.com/current
crontab -l -u deploy | grep artisan
sudo systemctl reload php8.4-fpm
php /var/www/example.com/current/artisan config:cache
Link this runbook from your Ansible playbooks for PHP server provisioning docs so new servers inherit the same paths.
Database and queue issues
Include read-only checks before mutating data. Show how to inspect slow queries without killing production:
mysql -e "SHOW FULL PROCESSLIST\G" | head -40
php artisan queue:failed --json | head
redis-cli INFO memory
For enterprise Laravel applications, add runbooks for payment gateway callbacks. Document how to replay a webhook safely and which idempotency keys prevent double charges.
Secrets and access
Runbooks must never store live passwords or API keys. Reference secret locations instead: "Retrieve DB password from Ansible Vault" or ".env on shared path." See Ansible Vault for secrets for encryption patterns. Rotate credentials through a separate procedure doc.
When Should You Update and Test Your Runbooks?
A runbook rots the day production changes without the doc changing. Treat updates as part of the deploy checklist, not a someday task.
Update triggers
- Every infrastructure change: new PHP version, new queue driver, CDN swap.
- After any incident where the runbook was wrong, missing, or untested.
- When onboarding a new on-call rotation member who got lost in a doc.
- Quarterly review calendar event, even if nothing broke.
On sites using GitLab CI and Deployer, I add a merge request checklist item: "Runbooks updated?" It takes thirty seconds and prevents drift. Testing and optimization work should include game-day exercises, not only load tests.
Game-day testing
Schedule a non-production drill. Inject failure, time the fix, note gaps. A table-top review helps, but live drills expose bad hostnames and expired SSH keys. Record the elapsed time and update the runbook with anything that slowed the engineer.
Pair drills with alerting docs. If Prometheus rules changed, read alerting with Prometheus Alertmanager and confirm each critical alert still maps to a runbook URL.
Measure usefulness
Track simple metrics: time to mitigate, number of escalations, runbook clicks during incidents. If engineers bypass the doc and SSH from memory, ask why. Usually the steps are wrong, too long, or hard to find.
The Atlassian runbook overview recommends tying documentation updates to post-incident reviews. Close the loop in your retro template: "Runbook created or updated? Y/N."
What Are Common Mistakes When Teams Write Runbooks?
Knowing what fails helps you write effective runbooks the first time. These show up on small teams and enterprise setups alike.
- Wall of prose: Paragraphs instead of numbered steps. Break into commands and expected output.
- Assumed context: "Fix the worker" without saying which systemd unit name production uses.
- No verification step: Engineer restarts a service and closes the ticket while checkout is still broken.
- Stale screenshots: UI moves faster than text. Prefer command-line checks that survive redesigns.
- Orphan docs: Runbook exists but no alert links to it. Wire the connection both ways.
- Secrets in plain text: Never. Reference vault paths and rotation procedures instead.
On a legal-tech portal I built, document upload failures needed a runbook that covered Spatie Media Library disk config and S3 credential rotation separately. Mixing them created confusion during a real outage. Split docs, clear titles, faster fixes.
For debugging workflows that feed into runbooks, see AI-assisted debugging. AI can draft a first pass, but a human must run every command on staging before production trust.
If you are building operational maturity into a new platform, custom software development should include runbook templates in the handover package. Clients with small teams especially need docs they can follow without calling you at midnight.
Examples of production systems that benefit from tight ops docs appear in our Adventure Third Pole Trek booking platform and other Laravel deployments with queue-heavy workloads.
Key Takeaways
- Write one runbook per failure mode with metadata, triggers, diagnosis, resolution, rollback, and escalation.
- Use copy-paste commands, expected output, and real production paths — not generic advice.
- Structure on-call docs in three tiers: quick fix, deeper diagnosis, then escalation.
- Link every critical alert to its runbook URL and update docs after deploys and incidents.
- Run game-day drills quarterly; a stale runbook is worse than no runbook because it breeds false confidence.
- Store runbooks in Git, review them like code, and never embed live secrets.
People Also Ask
What is the difference between a runbook and a playbook?
A runbook is a step-by-step technical procedure for a specific operational task or incident. A playbook is broader — it may cover roles, communications, and policy across many runbooks. In practice, teams use "runbook" for engineer-facing steps and "playbook" for org-wide incident response. Both should link together.
How long should a runbook be?
Long enough to fix the issue without guessing, short enough to scan in under three minutes. Most effective runbooks fit on one to two screens. Move deep background to linked architecture docs. Keep the incident path lean.
Who should write runbooks?
The engineer who last fixed the incident writes the first draft. A peer who did not respond to the outage reviews it for missing steps. Platform or DevOps owns the library, but application teams own service-specific procedures like payment replay or cache warming.
What tools store runbooks best?
Git-backed Markdown works for teams already using GitLab or GitHub. PagerDuty, Grafana IRM, and Confluence add alert integration and search. Choose based on where on-call already lives. The best tool is the one your night shift will actually open.
Ship Runbooks Before You Need Them
The best time to write effective runbooks is right after you fix something painful — while the steps are fresh and the logs still open. Start with your top five recurring alerts, use a shared template, test on staging, and wire each doc to its monitor. Your on-call future self will thank you.
Need help building reliable ops docs, deploy pipelines, or on-call readiness for a Laravel or WordPress production site? Review our web development services, browse the portfolio for production examples, or contact us to plan runbook coverage for your stack. For broader context on reliability culture, read about my production engineering work or explore related posts on the blog.
Frequently Asked Questions
A runbook is a step-by-step operational playbook for a specific failure or task. It tells an engineer what to do when an alert fires or a user report matches a known pattern—not general architecture documentation.
Long enough to fix the issue without guessing, short enough to scan in under three minutes. Most effective runbooks fit on one to two screens; move deep background to linked architecture docs.
The engineer who last fixed the incident writes the first draft. A peer who did not respond to the outage reviews it. Platform or DevOps owns the library; application teams own service-specific procedures like payment replay.
A runbook is a step-by-step technical procedure for one operational task or incident. A playbook is broader—it may cover roles, communications, and policy across many runbooks. In practice, teams use runbook for engineer-facing steps and playbook for org-wide incident response. Both should link together so on-call staff know when to follow technical steps versus wider comms and escalation policy.
Every runbook needs metadata and ownership at the top: severity, service name, owner team, and last-tested date. Include clear triggers, impact, prerequisites, numbered diagnosis steps, resolution commands with expected output, verification checks, rollback notes for mutating steps, and escalation paths with contact names and time limits. Copy-paste commands must use real hostnames, usernames, and PHP binary paths from production—not vague instructions like check the logs without a file path.
Structure for scanability under time pressure, not literary flow. Use a three-tier layout: Tier 1 is a quick fix of one to three steps that resolve most occurrences; Tier 2 covers deeper diagnosis with log paths, SQL queries, curl checks, and Redis pings; Tier 3 defines when to stop, what data to collect, and who owns root-cause work. Front-load action, bury theory, and cross-link related runbooks so chained incidents—deploy failures leading to opcache or queue issues—do not force wiki searching at 2 a.m.
Document stack-specific paths because generic advice fails when storage permissions or stale opcache after deploy is the root cause. Cover Deployer 7 symlink releases and cron jobs pointing at old release paths, PHP-FPM reload steps after deploy, queue worker restarts via systemctl and php artisan queue:restart, and read-only checks before mutating data such as SHOW FULL PROCESSLIST and php artisan queue:failed. For payment-heavy apps, add webhook replay procedures with idempotency key notes. Reference secret locations like shared .env paths instead of embedding credentials.
Update after every infrastructure change—new PHP version, queue driver swap, CDN change—and after any incident where the runbook was wrong, missing, or untested. Add a deploy checklist item asking whether runbooks were updated. Schedule quarterly reviews and game-day drills: inject failure in non-production, time the fix, and record gaps like bad hostnames or expired SSH keys. Confirm each critical Prometheus or Zabbix alert still maps to a runbook URL after alerting rule changes. Track time to mitigate and whether engineers bypass the doc.
Wall-of-prose paragraphs instead of numbered steps with expected output. Assumed context such as fix the worker without naming the systemd unit production actually uses. Missing verification so an engineer restarts a service while checkout remains broken. Stale screenshots that outlive UI changes—prefer CLI checks. Orphan docs with no alert linking back to them. Secrets stored in plain text instead of vault references. Mixing unrelated failure modes on one page, such as Spatie Media Library disk config and S3 credential rotation combined, which slows diagnosis during real outages.
Git-backed Markdown works well for teams already using GitLab or GitHub—you can keep runbooks in a docs/runbooks/ folder, render them in GitLab or Confluence, and review changes like code. PagerDuty, Grafana IRM, and Confluence add alert integration and search. Choose based on where on-call already lives. The best tool is the one your night shift will actually open when an alert fires, not the platform with the most features your team never bookmarks.
No. Runbooks must never store live passwords or API keys. Reference secret locations instead, such as retrieve DB password from Ansible Vault or credentials in .env on the shared deploy path. Rotate credentials through a separate maintenance procedure document. A broken command in a runbook is a production bug; embedding secrets creates a breach waiting to happen and forces doc redaction every rotation cycle.
Pair each runbook with the alert that should open it. If the alert and document live in different systems, link them explicitly in both directions—the alert configuration should include the runbook URL, and the runbook metadata should name the related alert such as QueueDepthHigh or DiskUsageCritical. After Prometheus or Alertmanager rule changes, confirm every critical alert still maps to a current runbook. Orphan docs that no monitor references rarely get used during incidents.
Game-day testing is a scheduled non-production drill where you inject a realistic failure, time the fix, and note documentation gaps. A table-top review helps, but live drills expose bad hostnames, wrong PHP binary paths, and expired SSH keys that calm reading misses. Record elapsed time and update the runbook with anything that slowed the engineer. Pair drills with alerting verification so monitors and docs stay aligned after infrastructure changes.
Symptom-led runbooks start from what the user or monitor sees, such as 502 Bad Gateway on checkout. Alert-led runbooks start from the monitoring rule name, such as DiskUsageCritical on prod-web-01. Procedure-led runbooks cover scheduled work like credential rotation, not live incidents. For Linux administration on Laravel stacks, mixing symptom-led and alert-led formats works because on-call engineers may arrive from either a user report or a PagerDuty page naming a specific alert.
Yes. A runbook rots the day production changes without the doc changing, and a stale runbook breeds false confidence—engineers follow wrong paths while believing they are safe. Wrong symlink paths after Deployer 7 releases, outdated PHP-FPM unit names, or missing queue restart steps can extend outages. Treat runbook updates as part of the deploy checklist, test quarterly with game-day drills, and close post-incident retros with whether the runbook was created or updated. Accurate docs shorten recovery; outdated ones waste precious minutes.
0 Comments
Leave a comment
Your email is not published. Comments appear once they have been read. Sign in to have your details filled in.
