On-Call and Incident Response Runbook

By Kokil Thapa | Last reviewed: August 2026

When a production Laravel application fails at 2 AM, the difference between a five-minute fix and a four-hour outage is usually documentation. An effective On-Call and Incident Response Runbook transforms panic into procedure by providing exact commands, decision trees, and escalation paths before the alert even fires. Without this structured knowledge base, engineers waste critical minutes guessing server paths or searching Slack history while users experience downtime.

I have maintained production web systems since 2010, and I can confirm that memory is not a valid operational strategy. Whether you are running a high-traffic WooCommerce store or a legal-tech portal like Laravel applications serving Nepal-based clients, fatigue will erase your mental notes during an emergency. The runbook bridges the gap between "I think I fixed this last time" and "Here is the exact command to restore service." This guide covers building a runbook specifically for PHP/Laravel stacks, grounded in real deployment workflows using Deployer 7, GitLab CI, and Ubuntu servers.

What should be included in an On-Call and Incident Response Runbook?

A functional runbook is not a generic wiki page; it is an executable checklist designed for high-stress environments. In my experience managing infrastructure for eCommerce platforms and legal service portals, every entry must pass the "3 AM test": can a tired engineer execute this safely without reading three other documents? For Laravel 12.x and modern PHP 8.4 stacks, your runbook needs five non-negotiable components.

1. Alert Definition and Severity Matrix: Clearly distinguish between P1 (site down, payment gateway failing) and P3 (admin dashboard slow). Define what constitutes a "wake-up" event versus a "morning triage" task.
2. Immediate Triage Commands: Copy-pasteable commands to check system health. Do not link to documentation; embed the actual artisan or systemctl commands.
3. Architecture Context: A simplified diagram showing how Nginx, PHP-FPM, Redis, and MySQL interact. When queues back up, engineers need to know if the bottleneck is the worker count or the database connection limit.
4. Rollback and Mitigation Procedures: Specific steps to revert the last deploy via Deployer or disable a feature flag. Mitigation (restoring service) always takes precedence over root cause analysis.
5. Escalation Contacts and Access: Verified phone numbers for senior engineers, hosting providers, and third-party API vendors. Include links to credential vaults, never passwords directly.

For teams managing multiple sites on shared infrastructure—a pattern I use for several sister sites deployed via GitLab CI—the runbook must also specify environment isolation. Confusing the staging database with production during an incident is a catastrophic but preventable error. Always prefix critical commands with explicit environment checks.

How do you write effective runbook commands for Laravel production debugging?

The most common failure mode in runbooks is vagueness. "Check the logs" is useless when the engineer does not know whether you use daily rotation, single-file logging, or syslog. For Laravel 12 applications running on Ubuntu 24.04 with PHP-FPM, precision saves time. Every command in your On-Call and Incident Response Runbook must be tested against the current production environment, not your local Valet setup.

Essential Diagnostic Commands

Include these exact commands in your runbook, adjusted for your specific Deployer release paths. On a standard zero-downtime deployment, the active release symlink is typically /var/www/project/current.

# Verify current release path and deployment timestamp
ls -la /var/www/project/current
cat /var/www/project/current/.env | grep APP_ENV

# Check Laravel-specific health endpoints (if configured)
curl -s https://example.com/up | jq .

# Tail application logs with context
tail -n 100 /var/www/project/current/storage/logs/laravel.log | grep -i "error\|exception"

# Inspect PHP-FPM process pool status
systemctl status php8.4-fpm
journalctl -u php8.4-fpm --since "10 minutes ago" --no-pager

# Check queue worker health and stuck jobs
php /var/www/project/current/artisan queue:monitor --timeout=60
redis-cli llen queues:default

# Validate database connectivity without running migrations
php /var/www/project/current/artisan db:show --counts=users,orders

A frequent mistake I encounter during CI/CD pipeline audits is outdated runbook commands referencing old PHP versions. If you upgraded from PHP 8.3 to 8.4 last month but the runbook still says php8.3-fpm, the on-call engineer will waste minutes troubleshooting a service that no longer exists. Version pinning in documentation is as important as version pinning in composer.json.

Queue and Cache Emergency Procedures

Laravel queues are often the first component to degrade under load. Your runbook needs explicit restart procedures that respect graceful shutdowns. Simply killing workers can corrupt job payloads.

# Gracefully restart all queue workers after deployment
php /var/www/project/current/artisan queue:restart

# Force-stop stuck workers (last resort only)
pkill -f "queue:work"
supervisorctl restart laravel-worker:*

# Flush application cache without clearing sessions
php /var/www/project/current/artisan config:clear
php /var/www/project/current/artisan route:clear
php /var/www/project/current/artisan view:clear

# Verify Redis memory pressure
redis-cli info memory | grep used_memory_human

Always document the expected output of healthy commands. If redis-cli llen queues:default normally returns 0-50 but currently shows 15,000, that is your smoking gun. Without baseline expectations, metrics are just noise.

How should incident severity levels be defined for PHP web applications?

Severity definitions must be tied to business impact, not technical symptoms. A 500 error on an admin report page is technically identical to a 500 error on the checkout endpoint, but operationally they are worlds apart. For eCommerce clients using WooCommerce or custom Laravel carts, I define severity based on revenue exposure and user-facing functionality.

This matrix prevents alert fatigue. If every notification triggers a P1 response, engineers stop responding to real emergencies. For Nepal-based businesses operating on tighter budgets, clearly defining P2 vs P3 helps manage expectations about after-hours support costs. A client paying Rs 15,000/month (~USD 110) for maintenance should understand that cosmetic fixes wait until Monday morning, while payment failures get immediate attention.

How do you handle incident communication and stakeholder updates?

Technical resolution is only half the job. Stakeholders do not care about your strace output; they want to know when service will restore and what they should tell customers. Pre-written communication templates eliminate the cognitive load of drafting messages while debugging. In my work with legal-tech clients where trust is paramount, transparent communication during outages preserves relationships more than pretending nothing happened.

Status Update Templates

Maintain these templates in your runbook with placeholders clearly marked. Adapt tone for your audience—internal teams get technical detail, clients get business impact.

## INITIAL ACKNOWLEDGMENT (within 15 min of P1)
Subject: [INCIDENT] Service degradation detected — {SERVICE_NAME}
Status: Investigating
Impact: {USER_FACING_SYMPTOM}
Started: {TIMESTAMP_NPT}
Next update: {TIMESTAMP + 30 MIN}

We are aware of {SYMPTOM} affecting {SCOPE}. 
Engineering is actively investigating. 
No ETA yet — we will update in 30 minutes.

## PROGRESS UPDATE (every 30-60 min)
Status: {INVESTIGATING | IDENTIFIED | MITIGATING | RECOVERING}
Current action: {WHAT_WE_ARE_DOING_NOW}
ETA: {ESTIMATE_OR_UNKNOWN}
Workaround: {IF_AVAILABLE}

## RESOLUTION NOTICE
Status: Resolved
Duration: {START_TIME} to {END_TIME} ({TOTAL_MINUTES} min)
Root cause: {ONE_SENTENCE_SUMMARY}
Prevention: {FOLLOW_UP_ACTION_PLANNED}

Service is fully restored. Post-incident review scheduled for {DATE}.

For Nepal-based operations, consider bilingual updates if your client's customer base includes non-English speakers. Also account for local business hours and festivals—an incident during Dashain may require different communication cadences than a regular Tuesday. Link these templates to your server security and monitoring documentation so the on-call engineer has full context.

What belongs in a post-incident review for Laravel applications?

The post-incident review (PIR), sometimes called a post-mortem, is where organizational learning happens. Without it, you will fix the same bug three times in six months. A PIR is not a blame session; it is a structured analysis of systemic failures. For Laravel applications, focus on actionable improvements to code, configuration, or process.

PIR Document Structure

• Timeline: Minute-by-minute account from first alert to full recovery. Use timestamps from monitoring tools, not memory.
• Impact Quantification: Orders lost, users affected, revenue estimate. For a WooCommerce florist site, "23 failed checkout attempts during peak Valentine's window" is more useful than "checkout was down."
• Root Cause Analysis: Use the "Five Whys" technique. "Queue failed" → Why? → "Redis OOM" → Why? → "Memory limit too low" → Why? → "Never tuned after traffic doubled" → Why? → "No capacity review process."
• Action Items: Concrete tasks with owners and deadlines. "Increase Redis maxmemory" is better than "monitor Redis better." Link each item to a Jira ticket or GitLab issue.
• Runbook Updates: What was missing or wrong in the current runbook? This section closes the feedback loop.

Schedule PIRs within 48 hours of P1 incidents while memories are fresh. For distributed teams across time zones, record the session for async review. The goal is updating the On-Call and Incident Response Runbook so the next incident is faster, cheaper, and less stressful.

Start Building Your Runbook Before the Next Outage

An On-Call and Incident Response Runbook is not a document you write once and archive. It is a living artifact that evolves with your application, your team, and your infrastructure. Start small: pick your top three recurring alerts and document them thoroughly this week. Test the commands during a quiet period. Share the draft with whoever covers your next on-call shift and ask what is missing. For teams managing Laravel applications, WooCommerce stores, or legal-tech platforms in Nepal or globally, this discipline compounds into reliability over months and years.

If your production environment lacks structured incident response procedures, or if your current runbook is outdated and untested, reach out through my contact page. I help teams build practical, maintainable operational documentation grounded in real production experience—not theoretical frameworks that collapse under pressure. Whether you need a full runbook audit, deployment pipeline hardening, or hands-on incident response training for your developers, we can scope an engagement that fits your operational reality and budget.