Running Blameless Post-Mortems

01/ The Tactical Resolution

The Case Study: The Friday Database Wipeout

The Problem

At 4:15 PM on a Friday, our production PostgreSQL database went offline. For three hours, the site was completely down. Our business users couldn't log in, payment processing failed, and we lost approximately $45,000 in checkout revenue.

During the incident response, it was discovered that a junior developer, Leo, had run a database cleaning script in the production console instead of the staging console. He had copied the wrong environment variables.

Once the database was restored from backups, a Slack flame-war broke out. A senior architect commented: “Why was a junior running raw scripts in production anyway? This was incredibly careless.” Leo was silent, terrified that he was about to be fired. The team was on edge, dreading the upcoming Monday incident review meeting. If the review became a trial of Leo's mistake, developers would start hiding errors, refusing to deploy code on Fridays, and avoiding any task that carried operational risk.

The Playbook: The System-First Inquest

A blameless post-mortem assumes that well-intentioned people make mistakes because the system allowed them to do so. If a junior can wipe the production database with one command, the problem is the permission architecture, not the junior.

Step 1: Set the Blameless Ground Rules

At the start of the post-mortem meeting, read this statement aloud:

• The Prime Directive: "We assume that everyone did the best job they could, given what they knew at the time, their skills, and the resources and tools available. We are here to fix our architecture and processes, not our people."
• The Action: Ban the words "careless," "mistake," or "human error" from the document. Focus on what allowed the action to happen.

Step 2: The "5 Whys" Incident Chain

Trace the incident backward using system-level questions:

| Question Chain | Answer | Action Item / Mitigation |
| :--- | :--- | :--- |
| **Why did the site go down?** | The production checkout table was dropped. | Set Postgres table deletion locks. |
| **Why was the table dropped?** | Leo executed a clean-up script in production. | Remove direct production DB write access. |
| **Why was he running it there?** | Staging and Production credentials looked identical. | Standardize distinct prompt colors for terminal shells. |
| **Why was the script written?** | To clean up test transactions manually. | Build an automated cleanup cron job for staging. |

By focusing on the "Why," the discussion shifts from “Leo shouldn't have run that” to “We must automate staging cleanup and restrict write access to production.”

Step 3: Publish the "Post-Mortem artifact"

Document the incident in a centralized wiki using a standard template. Include:

1. The Timeline: Exact timestamps of detection, escalation, diagnosis, and resolution.
2. The Root Cause: The systemic failures that allowed the incident to happen.
3. The Corrective Actions: Jira tickets with assigned owners and 14-day completion deadlines.

• The Cultural Standard: Publish this document to the whole company. Show that we treat outages as free lessons in engineering resilience.

The Long-Term Impact

• System Security: We completely locked down production DB write permissions, requiring multi-party approval keys for migrations.
• Psychological Safety: Leo didn't quit. Instead, he co-authored our database access policy and became one of our strongest advocates for system security, mentoring other juniors on staging safety.
• MTTR Reduction: Because the team no longer feared blame, the next minor incident was reported immediately by a developer within 2 minutes of occurrence, allowing us to resolve it in under 10 minutes (a 50% decrease in resolution time).