1. Response times against your contract
We show the slowest case as well as the typical one, because an average can hide a bad night.
| Measure | Contract | Typical | 3 in 4 within | Slowest |
|---|---|---|---|---|
| Acknowledge | 15 min | 3 min | 5 min | 11 min |
| Cause and severity | 30 min | 12 min | 16 min | 26 min |
| Your engineer picks up an escalation | Agreed: 20 min | 8 min | 14 min | 41 min |
One escalation on 14 September took 41 minutes to be picked up, because the page went to an engineer on leave. Your rota has since been updated.
2. Alert quality
- Fixed by an approved runbook
- 37
- Noise
- 16
- False alarms, explained in the ticket
- 8
- Escalated to your engineer
- 23
- Closed without waking you: 61 of 84. 37 known issues fixed by an approved runbook, 16 noise and 8 false alarms.
- Escalated to your engineer: 23. Each went with a timeline, the evidence and a likely cause.
- Pages nobody needed to act on: 24 of 84, or 29%, down from 38% in July. The quarterly goal is 23% or lower.
- Incidents your users spotted before an alert did: 1 of 6. Password reset emails stopped sending and no alert covered it. A synthetic check (a scripted test of the journey, run every few minutes) now does.
- Alerts per incident: 4.2, down from 6.8. Fewer duplicate pages for the same problem.
3. Incidents
| Date | Service | What users saw | Duration | Report |
|---|---|---|---|---|
| 3 Sep | checkout-api | Slow checkout for about 1 in 5 users | 17 min | Published 5 Sep |
| 9 Sep | search | Empty search results in one region | 42 min | Published 12 Sep |
| 14 Sep | notifications | Password reset emails not sent | 2 h 10 min | Published 17 Sep |
| 21 Sep | checkout-api | Errors on card payments | 9 min | Published 23 Sep |
| 26 Sep | admin | Admin pages slow for staff | 25 min | Published 29 Sep |
| 29 Sep | images | Product images missing | 14 min | Draft, due 2 Oct |
Each report covers the timeline, who was affected, the contributing factors, what went well, where we got lucky and what we told your customers. The search incident on 9 September had three contributing factors: a config change, a missing check in the release pipeline and an alert that only watched error rates, not empty results.
4. Runbooks
| Runbook | Runs | Worked | Last tested |
|---|---|---|---|
| Roll back checkout-api release | 2 | 2 | 12 Aug |
| Scale out search workers | 7 | 7 | 12 Aug |
| Restart image-resize worker | 28 | 28 | 2 Sep |
5. Reliability targets
| Service | Target (30 days) | Actual | Allowance used | Status |
|---|---|---|---|---|
| Checkout succeeds | 99.9% | 99.94% | 60% | On target |
| Search returns results in under 500 ms | 99% | 98.7% | 130% | Over budget |
Search used more than its monthly allowance for failure, mostly in the 9 September incident. We recommend pausing risky search releases for two weeks while it recovers; you decide whether to.
6. Reliability goals this quarter
- Cut pages nobody needed to act on to 23% or lower by 31 December. Now 29%. On track.
- Add a reliability target for sign-up by 30 November. Draft agreed with your product lead; measurement starts 1 November. On track.
7. Follow-up actions
| Action | Owner | Due | Status |
|---|---|---|---|
| Add an empty-results check to search releases | Your team | 15 Oct | Open |
| Synthetic check for password reset | Tracefox | 20 Sep | Done |
| Fix image-resize memory leak | Your team | 30 Nov | Open |
| Remove 6 duplicate disk alerts | Tracefox | 30 Sep | Done |
8. Changes we made, and what’s coming
- Raised the disk alert on the reporting database from 80% to 90% full, because it paged 9 times without anyone needing to act.
- Removed 6 duplicate disk alerts and added the password reset check.
- Your planned move of search to a new region on 18 October: we’ll watch it closely and have the rollback runbook tested beforehand.
9. Near misses
On 25 September the TLS certificate for the payments callback was 6 days from expiring when our certificate alert caught it. Your team renewed it the same day. No users were affected; we list near misses because they show which defences are doing the work.