September 11, 2026
13 min read
By Kokil Thapa | Last reviewed: September 2026
Production breaks at the worst moment. Payment callbacks fail during a holiday sale. A law-firm portal returns 500 errors while a client uploads documents. Your monitoring pings at 2 a.m. and nobody knows who owns the fix. Incident Response: A Practical Playbook turns that chaos into a repeatable sequence: detect, triage, contain, recover, and document. This guide is written for developers, founders, and small ops teams running Laravel, WordPress, WooCommerce, or custom PHP on Linux—exactly the stack I maintain on client projects in Nepal and abroad. If you already have an on-call and incident response runbook, treat this page as the field manual that sits beside it.
Quick answer: Incident Response: A Practical Playbook means defining severity levels, assigning a single incident commander, isolating the blast radius, restoring service from known-good backups, and writing a blameless postmortem within 48 hours—before you tune architecture or buy new tools.
What is Incident Response: A Practical Playbook for web applications?
An incident is any event that threatens availability, integrity, or confidentiality of a system users depend on. That includes full outages, partial degradation, data corruption, credential leaks, and payment webhook failures that leave orders stuck in limbo.
A playbook is not a 40-page PDF nobody reads. It is a short set of checklists your team can execute under stress. In practice, the best playbooks fit on one wiki page plus three severity-specific runbooks.
For a production Laravel application, incidents usually cluster into a handful of categories:
- Application errors — PHP fatal errors, queue worker death, failed deploys, opcache serving stale code after symlink swap.
- Infrastructure failures — disk full, MySQL connection exhaustion, Redis memory cap, SSL expiry, DNS misconfiguration.
- Security events — brute-force spikes, uploaded malware, leaked API keys, suspicious admin logins.
- Integration failures — payment gateway timeouts, SMS provider outages, third-party API auth rotation.
- Data incidents — bad migration, accidental DELETE, duplicate charges, corrupted uploads.
Your playbook should name these categories upfront. When the pager fires, the first question is not "what broke?" but "which category—and what severity?"
The NIST Computer Security Incident Handling Guide (SP 800-61 Rev. 2) formalises similar phases. You do not need enterprise SOC tooling to apply the model. A three-person agency in Kathmandu can run the same sequence with UptimeRobot, server logs, and a shared Slack channel.
How do you classify incident severity before you start fixing?
Severity decides who wakes up, how fast you communicate, and whether you roll back or hot-patch. Without tiers, every alert feels like a fire drill. Teams burn out. Real emergencies get treated like routine noise.
I use four levels on production systems I maintain. Adjust names to match your business, but keep the boundaries crisp.
| Level | Definition | Example | Response target |
|---|---|---|---|
| SEV-1 | Complete outage or active data breach | Site down, payment double-charges, leaked client documents | Acknowledge in 5 min; exec + client comms |
| SEV-2 | Major feature broken for most users | Checkout fails, login broken, booking form 500s | Acknowledge in 15 min; hourly updates |
| SEV-3 | Partial degradation or workaround exists | Slow queries, one payment method down, stale cache | Fix in business hours; daily note |
| SEV-4 | Minor bug, no user impact yet | Log noise, non-critical cron warning | Backlog ticket |
Assign one incident commander
Every SEV-1 and SEV-2 incident needs a single incident commander (IC). The IC does not fix everything personally. They coordinate, assign tasks, and own the timeline. On small teams, the senior developer on call becomes IC by default.
The IC's first three actions take under ten minutes:
- Open a dedicated incident channel or thread. Name it with date and severity, for example
2026-09-11-sev2-checkout. - Post the current user impact in plain language. "Customers cannot complete Khalti payments" beats "500 on PaymentController."
- Assign roles: one person investigates, one communicates, one watches for side effects.
Legal-tech portals raise the stakes. A document upload failure on a client portal is SEV-2 at minimum. An unauthorised download of case files is SEV-1 plus legal review. I've shipped portals like Mijar Law Associates and Notary Nepal where uptime and document confidentiality are contractual expectations, not nice-to-haves.
What should you prepare before any production incident happens?
Most incident pain comes from missing preparation, not missing talent. You cannot invent backups during an outage. You cannot grep logs you never retained.
Build a one-page contacts and access sheet
Store it in three places: password manager, printed copy, and offline phone note. Include hosting provider login, DNS registrar, payment gateway support numbers, SSL renewal dates, and on-call rotation. For Nepal payment integrations, keep eSewa, Khalti, and ConnectIPS merchant support lines handy.
Verify backups and recovery paths monthly
A backup you have never restored is a hope, not a plan. On MySQL 9.7 or MariaDB 12.3, run a test restore to a staging instance. For PostgreSQL 18, follow a documented point-in-time recovery playbook at least once per quarter.
# Quick MySQL restore smoke test (staging only)
mysql -u root -p staging_db < /backups/nightly/app_2026-09-10.sql
php artisan migrate:status
php artisan queue:restartCentralise logs before you need them
Scattered logs across three servers make root-cause analysis slow. Even a small team benefits from shipping Apache, PHP-FPM, Laravel, and MySQL slow-query logs to one searchable place. Start with the approach in log aggregation for small teams. Pair it with AI-powered log analysis only after basic retention works.
Document rollback for every deploy path
If you use Deployer 7 with symlinked releases, rollback is one command:
dep rollbackKnow the exact release directory, PHP-FPM reload step, and queue restart sequence. After symlink swap, opcache may serve old bytecode until PHP-FPM reloads. That mismatch causes "works on previous release" confusion during incidents.
Professional Linux system administration and support and maintenance contracts should explicitly include backup verification and runbook updates—not just uptime promises.
How do you contain and recover during a live production incident?
Containment stops bleeding. Recovery restores normal service. Teams that skip containment often fix one symptom while making another worse—clearing cache while a bad deploy still serves traffic, for example.
Containment checklist for web applications
- Enable maintenance mode if the site is actively corrupting data or leaking information.
- Block attack traffic at the firewall or CDN when you see brute-force or scraping spikes.
- Disable the broken integration via feature flag or env toggle rather than deleting code mid-incident.
- Revoke compromised credentials—API keys, admin sessions, Sanctum tokens—before deeper forensics.
- Snapshot evidence—copy relevant logs and database rows before cleanup destroys audit trail.
Laravel maintenance mode on PHP 8.3+ or 8.5:
php artisan down --secret="incident-2026-09-11" --retry=300The secret URL lets staff verify fixes while public users see a maintenance page. Document the secret in the incident channel only.
Common Laravel and PHP recovery moves
These fixes appear repeatedly on production Laravel 12 and Laravel 13 applications:
- Queue backlog — restart workers:
php artisan queue:restartthen confirm Supervisor restarted processes. - Config cache stale after .env change —
php artisan config:clear && php artisan config:cache - Permission errors after deploy — ensure
storage/andbootstrap/cache/are writable by PHP-FPM user. - 502 from PHP-FPM — check
pm.max_childrenexhaustion; temporary relief by raising pool size, permanent fix by query optimisation. - Disk full — rotate logs, clear old releases, expand volume; never delete MySQL binlogs without understanding replication impact.
For WordPress 7.1 or WooCommerce 11.1 shops, disable the offending plugin via filesystem rename before wp-admin loads. That beats editing the database under pressure.
Status updates that reduce client panic
External updates need four fields every time: what users experience, what you know, what you are doing, and when you will update next. Avoid technical jargon in client-facing messages. "We are restoring the booking system from last night's backup" is enough.
DNS and SSL incidents confuse non-technical owners. If propagation or certificate renewal is the root cause, point stakeholders to how DNS resolution affects perceived downtime. SSL expiry is a SEV-1 preventible event—monitor it 30 days ahead.
How do you handle security incidents differently from outages?
Security incidents add legal and reputational dimensions. Speed still matters, but evidence preservation matters equally. The OWASP Incident Response project emphasises documenting chain of custody for any data you might need in a dispute or regulatory inquiry.
Immediate security response steps
- Confirm the event is real—not a scanner hitting a honeypot URL.
- Isolate affected systems without destroying logs. Snapshot disk or copy auth logs first.
- Force password resets for compromised admin accounts. Rotate API keys stored in
.env. - Check upload directories and
storage/app/publicfor unexpected PHP files. - Review recent git commits and deployment history for unauthorised changes.
On eCommerce systems like the Quick And Easy Nepalese Grocery platform, also audit order and payment tables for anomalies during the attack window. Duplicate captures and refund gaps show up in SQL before they show up in support tickets.
Rate limiting and abuse prevention should already be in place. If brute force triggered the incident, review your API rate limiting strategy after recovery—not six months later.
Use a JSON formatter to inspect webhook payloads and API responses during forensic review. Malformed or replayed callbacks often hide in verbose gateway JSON.
What belongs in a post-incident review and how do you prevent repeats?
Close every SEV-1 and SEV-2 incident with a blameless postmortem within 48 hours. Memory fades fast. The goal is systemic improvement, not finding who clicked the wrong button.
Postmortem template that teams actually fill in
Incident: [short title]
Severity: SEV-[1-4]
Duration: [start] to [resolved]
Impact: [users affected, revenue, data]
Timeline (UTC + local):
- HH:MM — alert fired
- HH:MM — IC assigned
- HH:MM — root cause identified
- HH:MM — service restored
Root cause:
[One paragraph, factual]
Contributing factors:
- [Missing monitor, no rollback test, etc.]
What went well:
- [Fast rollback, clear comms]
Action items (owner + due date):
1. [ ] Add disk-space alert — @devops — 2026-09-18
2. [ ] Automate post-deploy queue restart — @backend — 2026-09-25Consider automating incident postmortems with AI to draft timelines from Slack exports. A human must still validate facts before publishing internally.
Feed action items into your normal delivery pipeline. A postmortem that ends with "we should add monitoring" and no ticket is wasted paper. Tie remediation to CI/CD pipeline checks where possible—deploy gates catch repeat deploy failures before production.
Performance-related incidents deserve a separate track. Slow queries and memory leaks are incidents with a long fuse. Route them through testing and optimization and speed optimization workstreams so they do not reopen every peak season.
How do small teams run this playbook without a dedicated SRE?
Most Nepal agencies and product teams run production with two to five people. You do not need 24/7 follow-the-sun coverage on day one. You need clear ownership and realistic response windows.
Start with these minimum viable practices:
- One on-call person per week with phone notifications enabled.
- Uptime checks on homepage, login, checkout, and one API health endpoint.
- Nightly database backups with 7-day retention minimum; 30 days for legal and financial data.
- A shared runbook wiki linked from the services overview page your clients can reference.
- Quarterly game-day: simulate disk full or database restore on staging.
Enterprise clients building custom platforms benefit from baking incident hooks into architecture early. Enterprise application development should include audit logs, health endpoints, and feature flags—not bolt-on afterthoughts.
On sister sites I maintain with Deployer 7 and GitLab CI—legal portals sharing the same EC2 infrastructure—incident response is faster because deploy, rollback, and log paths are identical. Standardisation beats heroics.
If you run Adventure Third Pole Trek-style booking systems or high-traffic WooCommerce florists like Petals Qatar, map peak seasons on your calendar. Increase monitoring sensitivity before Dashain and Valentine's Day, not after the first outage.
For broader context on operational maturity, read the companion pieces on Ansible playbooks, infrastructure as code with Terraform, and the main blog index. Our portfolio shows live systems where these practices matter daily.
Key Takeaways
- Define SEV-1 through SEV-4 before the pager fires; assign one incident commander for every major event.
- Prepare contacts, tested backups, centralised logs, and a documented rollback path—preparation prevents panic.
- Contain first (maintenance mode, credential rotation, traffic blocks), then recover; snapshot evidence before cleanup.
- Security incidents need log preservation and payment or document audits, not just a quick redeploy.
- Close with a blameless postmortem and dated action items within 48 hours; update the runbook every time.
- Small teams win with standardised deploy paths, quarterly restore drills, and seasonal monitoring adjustments.
People Also Ask
What is the first thing to do when a production website goes down?
Confirm the outage is real with an external check, assign an incident commander, post severity and user impact in a dedicated channel, and only then investigate. Avoid pushing untested fixes while customers still hit the broken deploy.
How often should you test database backups?
Test a full restore monthly on staging and document the steps. Verify backup integrity after major schema migrations or infrastructure moves. A restore that takes four hours in calm conditions will take longer during a SEV-1.
Do you need a formal incident response plan for a small business website?
Yes, but it can fit on one page. Severity definitions, on-call contact, hosting and DNS logins, backup location, and rollback commands cover most small-business scenarios. Complexity grows with payment data, client documents, and multi-server setups.
What is the difference between an incident and a bug?
A bug is a defect in code or configuration. An incident is an active or recent event causing user impact or security risk. A latent bug becomes an incident when deploy, traffic, or an attacker triggers it in production.
Build your incident capability before the next outage
Incident Response: A Practical Playbook works when preparation, roles, and postmortems are routine—not emergency inventions. Start with severity tiers and a one-page runbook this week. Test one backup restore. Name your on-call rotation. The next production surprise will arrive; the only variable is whether your team has a sequence to follow.
If you want help hardening production Laravel apps, legal-tech portals, or eCommerce platforms with monitoring, backups, and runbooks, contact us or explore web development services. You can also read more about the author on the about page and see how these practices show up across our customer reviews.
Frequently Asked Questions
It is a short, executable set of checklists—not a long PDF—that turns production chaos into a repeatable sequence: detect, triage, contain, recover, and document. It applies to Laravel, WordPress, WooCommerce, and custom PHP on Linux, naming incident categories upfront and assigning severity before deep debugging begins.
Any event threatening availability, integrity, or confidentiality that users depend on. The article groups them into five categories: application errors such as PHP fatals and failed deploys, infrastructure failures like disk full or SSL expiry, security events including leaked API keys, integration failures such as payment webhook timeouts, and data incidents like bad migrations or duplicate charges.
Use four crisp tiers. SEV-1 is complete outage or active data breach—acknowledge in 5 minutes. SEV-2 is major feature broken for most users—15 minutes. SEV-3 is partial degradation with a workaround—fix in business hours. SEV-4 is minor with no user impact yet—backlog it. Without tiers, every alert becomes a fire drill.
Every SEV-1 and SEV-2 incident needs one single incident commander. The IC coordinates rather than fixing everything personally, opens a dedicated channel, posts user impact in plain language, and assigns investigator, communicator, and scribe roles within ten minutes.
Build a one-page contacts and access sheet stored in three places, verify backups with monthly test restores on MySQL 9.7, MariaDB 12.3, or PostgreSQL 18, centralise Apache, PHP-FPM, Laravel, and MySQL logs into one searchable place, and document rollback for every deploy path including Deployer 7 symlink releases and PHP-FPM reload steps.
Preparation, detection, containment, recovery, and post-incident learning. The model aligns with the NIST Computer Security Incident Handling Guide SP 800-61 Rev. 2, but a three-person agency can run the same sequence with UptimeRobot, server logs, and a shared Slack channel—no enterprise SOC tooling required.
Contain first: enable Laravel maintenance mode if data is corrupting, block attack traffic at the firewall, disable broken integrations via feature flags, revoke compromised credentials, and snapshot evidence before cleanup. Then recover using known-good backups, Deployer 7 rollback, queue restarts, config cache clears, and permission fixes on storage and bootstrap/cache directories.
Run php artisan down with a secret URL and retry header, for example php artisan down --secret="incident-2026-09-11" --retry=300. Public users see a maintenance page while staff verify fixes via the secret path. Document the secret only in the incident channel, not in client-facing updates.
Restart queue workers with php artisan queue:restart and confirm Supervisor restarted processes. Clear stale config cache after .env changes. Fix permission errors on storage and bootstrap/cache for the PHP-FPM user. Check PHP-FPM pm.max_children exhaustion for 502 errors. Rotate logs and clear old releases when disk is full, never deleting MySQL binlogs without understanding replication impact.
Disable the offending plugin via filesystem rename before wp-admin loads. That is faster and safer than editing the database under pressure during a live incident. This approach works when a plugin update or conflict causes checkout or admin failures without requiring database access mid-crisis.
Security incidents add legal and reputational stakes where evidence preservation matters equally with speed. Confirm the event is real, isolate without destroying logs, snapshot disk or copy auth logs first, force admin password resets, rotate API keys in .env, check upload directories for unexpected PHP files, and review recent git commits. On eCommerce systems, audit order and payment tables for duplicate captures during the attack window.
Close every SEV-1 and SEV-2 with a blameless postmortem within 48 hours. Include incident title, severity, duration, user impact, a UTC and local timeline from alert to resolution, root cause, contributing factors, what went well, and action items with owners and due dates. Feed action items into your delivery pipeline—a postmortem ending with no ticket is wasted effort.
Four fields every time: what users experience, what you know, what you are doing, and when you will update next. Avoid technical jargon. Saying you are restoring the booking system from last night's backup is enough. For DNS or SSL root causes, explain propagation or renewal delays in plain language since non-technical owners often misread those as continued breakage.
Most Nepal agencies run production with two to five people. Start with one on-call person per week, Uptime checks on homepage, login, checkout, and one API health endpoint, nightly database backups with 7-day minimum retention or 30 days for legal and financial data, a shared runbook wiki, and quarterly game-days simulating disk full or database restore on staging. Standardisation across identical deploy and log paths beats heroics.
On client portals where document upload failures block contractual workflows, treat upload failures as SEV-2 minimum. Unauthorised download of case files is SEV-1 plus legal review. On eCommerce platforms, payment webhook failures leaving orders stuck and duplicate charges during peak seasons like Dashain or Valentine's Day warrant SEV-1 or SEV-2 classification with faster acknowledgement and hourly client communication.
0 Comments
Leave a comment
Your email is not published. Comments appear once they have been read. Sign in to have your details filled in.
