{"id":2127,"date":"2026-10-19T18:10:00","date_gmt":"2026-10-19T18:10:00","guid":{"rendered":"https:\/\/askdoctorlive.com\/blog\/?p=2127"},"modified":"2026-10-19T18:10:00","modified_gmt":"2026-10-19T18:10:00","slug":"gitops-prune-incident-clinic","status":"publish","type":"post","link":"https:\/\/askdoctorlive.com\/blog\/gitops-prune-incident-clinic\/","title":{"rendered":"The Night the Appointments Vanished: A GitOps Story"},"content":{"rendered":"

Clinical teams already know the shape of the story we are about to tell. A routine step, performed in a hurry, exposes a gap in the protocol; the patient \u2014 in this case, a patient portal \u2014 is unharmed because the team followed their version of a time-out, and the protocol is rewritten so the gap cannot open twice. The team in question is a fictional but faithful composite: three people running the digital side of a small multi-specialty clinic network, their booking system and clinical content living on infrastructure managed through GitOps. The names are invented and the details are drawn from public runbooks rather than any real clinic, but the failure mode itself is uncomfortably ordinary. What happened to them one Tuesday is the best teaching case we know for why deletion needs a protocol of its own \u2014 and why automation, for all its virtues, hands the hardest judgement calls back to people.<\/p>\n

The Clinic’s GitOps Setup, in Plain Terms<\/h2>\n

Their arrangement was orthodox: every configuration described declaratively in a repository, an automated reconciler pulling that description and keeping the live environment aligned, changes arriving only through reviewed merges. Three services ran in production \u2014 appointments, results, and a content service serving guidance ranging from bronchitis symptoms to clinic policies<\/a>. Each service had its own data volume. Reconciliation had been quiet for weeks, which is precisely the condition under which people stop rehearsing for its failure.<\/p>\n

Tuesday, 02:14: The Finding<\/h2>\n

A developer merging a routine update to the appointments service had resolved a conflict badly. The resolution silently dropped that service’s manifest from the directory the reconciler watched in production. Nothing errored. The merge simply described a world in which the appointments service did not exist \u2014 and the reconciler, faithful as designed, made the live system match the description. It removed the service and, because the underlying storage carried a deletion reclaim policy, it removed the appointments data volume with it. The portal’s booking page went blank at a few minutes past two in the morning.<\/p>\n

Worth underlining: no outsider attacked anything. The system worked exactly as configured, and the configuration had a hole shaped like a dropped file. Every safeguard that mattered had to come from the people, which is how medicine would tell the story too.<\/p>\n

The GitOps Recovery, Step by Step<\/h2>\n

What follows is the six-step sequence the team actually worked through, lightly retold. Clinics will recognise the skeleton: stop, assess, treat, rehabilitate, resume, prevent.<\/p>\n

    \n
  1. Stop the automation.<\/strong> Before anything else, reconciliation was suspended so the recovering system could not be “corrected” back into the damaged state. Their tool’s suspension survives outside the repository \u2014 one less thing the broken merge could undo. Teams on other platforms need to be careful here: without an explicit override declared, a parent application can quietly re-enable what you disabled.<\/li>\n
  2. Assess, with the chart open.<\/strong> The team reconstructed events from three sources \u2014 the application’s own history, the reconciler’s event log, and the repository’s record of merges. The dropped file showed up quickly once the three timelines were laid side by side. Guesswork was unnecessary; the records were sufficient, which is the quiet reward for keeping records.<\/li>\n
  3. Treat the configuration.<\/strong> A revert went up as a reviewed pull request restoring the manifest, and was approved by the colleague who had not written it. Normal doorway, normal checks \u2014 an incident is the worst possible time to bypass the process that prevents incidents.<\/li>\n
  4. Rehabilitate the data.<\/strong> Configuration restored does not mean information restored. The appointments data came back from the overnight backup, restored into the re-created volume. The gap between the last backup and the incident was accounted for and reconciled by hand against the phone log. Data restoration is where the schedule risk lives, not the file recovery.<\/li>\n
  5. Resume, watching.<\/strong> Reconciliation was switched back on and the service observed through a full booking cycle before anyone declared the patient stable. The team also verified the content service and the results service \u2014 neighbours, in infrastructure as in wards, often share the injury.<\/li>\n
  6. Rewrite the protocol.<\/strong> The last step is the one that distinguishes a learning organisation from a lucky one, and it is the next section of this story.<\/li>\n<\/ol>\n

    What the Clinic Changed Afterwards<\/h2>\n

    Three permanent changes followed, each mapped to the failure that made it obvious. First, deletion protection: the appointments namespace and every data volume were marked so that automated cleanup could never remove them, and the storage policy was switched so that removed volumes outlive the services that used them. Second, a pre-merge check: their pipeline now inspects every change and raises a loud flag when files describing namespaces, data volumes or custom resource definitions disappear from a production directory \u2014 the exact signature of this incident, caught before merge rather than after. Third, a drill: the team rehearsed the suspension-and-restore sequence on the staging copy, on a weekday afternoon, with coffee. The real event had cost them a night; the rehearsal cost an hour and made the next one cheaper still.<\/p>\n

    Equally instructive is what did not happen afterwards. Nobody proposed “being more careful” as a policy, and nobody blamed the reconciler for doing what it was configured to do. The post-incident review spent its hour on the system: which guard was missing, where the deletion policy was too permissive, and why the merge review had not surfaced a removed file. Blame found no purchase because the protocol had a hole, and holes are repairable in ways that scapegoats are not. Clinics will recognise that stance; it is how they have run mortality reviews for decades.<\/p>\n

    For the pocket version \u2014 the whole protocol on one card, pinned above the desk where the next tired reviewer will actually see it:<\/p>\n