This is a story about software administration, not about medicine. It is invented, assembled around a scenario found in the Ops Atlas, and if it resembles a real team that is by chance. The team is small: a lead who handles operations, two developers and a part-time coordinator who keeps the support inbox tidy. They maintain the website and appointment-request forms for an online health information service. None of their work touches clinical decisions; it is forms, pages and background jobs that run in containers on a single server.
Monday: the red wall
On Monday morning the build dashboard was red. Not one job, all of them. The log lines were unfriendly: requests to the image registry had come back with a “too many requests” error. The developers assumed someone had broken the build configuration, and spent an hour comparing files. The lead, who had read a little about the discipline, suggested a quieter explanation.
Here is where the story borrows from the Atlas. The entry on ContainerOps in the Ops Atlas describes a scenario, labelled illustrative, in which thirty build runners pull images anonymously, around 180 times an hour, against a public registry whose anonymous allowance is 100 pulls per six hours. The arithmetic is brutal. The runners hit the wall and every build fails with a 429 response. Our team had fewer runners but a lot of rebuilds, because someone had asked for a visual refresh of the whole site that week.
Tuesday: the fix, and why it works
The Atlas offers a two-part fix: authenticate and add a pull-through cache. Signing in raised the team’s allowance, since the Atlas says that a free authenticated personal account receives 200 pulls per six hours, double the anonymous figure, and paid plans are effectively unlimited under fair use. The cache was a bigger decision. A pull-through cache sits between the builders and the public registry, keeps copies of the layers it has already fetched and serves them locally, so repeated builds stop hammering the source.
The lead was careful to note a line in the entry that stuck: the cache, it says plainly, offers no backup. If the original image vanished from the public registry, the cache might still hold it, but nobody should plan around that. The team resolved to keep its own copies of the handful of base images the site depended on.
Tuesday evening: the conversation nobody enjoys
Late in the day the lead gathered everyone for a short, slightly awkward chat. Nobody had done anything wrong, she said, but the week had exposed a habit: the team treated the registry as a tap that never runs dry. Real services have budgets, she explained. A public registry gives a certain number of pulls to anonymous users and a different number to signed-in ones, and no one had ever looked up the numbers. They wrote the limits into the inventory page so they would be visible to the next person who joined.
That small act turned out to matter more than the cache. Once a limit is written down, it becomes a design input. The developers began asking how many pulls a full rebuild needs, and whether a rebuild really had to start from scratch each time. They found two builds that fetched the same large base image twice in a row, and fixed that with a single line of configuration.
Wednesday: what else was fragile?
With builds running again, the coordinator asked the question that turns an incident into progress: what else could surprise us? The team made a list, and it was embarrassingly easy to fill.
- Several images used the latest tag, so nobody could say which version was live.
- No one knew who owned the three old background jobs.
- Containers had no memory limits.
- Logs were growing without any rotation on the server.
Every item on that list also appears in the Atlas as a recognised weakness. The team was not unusual; it was ordinary, and that was the reassuring part.
Thursday: a modest plan
The lead did not propose a revolution. She borrowed the structure of the Atlas thirty-day starting plan and trimmed it for a team of four: take an inventory, set log defaults, set up the mirror and switch a Dockerfile linter on in warning mode. In the second month, deploy by digest, let a bot propose base-image updates and run the most important services as an ordinary user with limits. In the third month, make linting blocking, agree a target for how quickly a fix should reach production and rehearse a rebuild from scratch.
The coordinator volunteered to keep the inventory because she was the one who liked spreadsheets. It became a living page with the image name, owner, source, last rebuild date and the date of the next review. Nothing fancy, but when a registry change came up later that quarter, they could check their exposure in two minutes.
Friday: the reaction from the team
One developer admitted to being sceptical. Digest pinning, he said, sounded like a promise to never update anything. The lead pointed to the Atlas, which answers that objection directly: pinning freezes fixes only when no bot is moving the pin. With a bot opening update proposals, the team receives small, reviewable changes instead of surprises. That changed his mind more than any lecture could.
The other developer asked about ports, having read that published ports can skip the host firewall. They checked and found one internal admin page reachable from everywhere. Binding it to the local address took two minutes. Both developers left the week feeling a little embarrassed and a lot more informed.
The cost, honestly counted
It is fair to ask what all this cost. In hours, about three days of the lead’s time spread over the quarter, a day and a half each from the developers and a few evenings of spreadsheet gardening from the coordinator. In money, close to nothing: the authenticated registry account was free and the cache ran on the same server. The savings are harder to count, because they consist of Monday mornings that did not go wrong. Still, the team could point to one number. Builds that used to fail a few times a week now failed only when the code itself was broken, which is the only kind of failure a build should report.
A few months later
The team now measures four things on a monthly page: what share of running containers use digests, how old the base images are, how many containers were killed for running out of memory and how long it takes a fix to land. The numbers are unspectacular and, most importantly, boring. The coordinator likes to say that boring dashboards are the goal.
The group also learned to share good habits outside the engineering room. When the support inbox needed a better routine for shared costs between volunteers, the coordinator pointed to a clear post on splitting bills with a recurring payment tracker as inspiration for how to document rules plainly. When they wanted a short description of the service’s own aims, they linked internally to an overview of how the platform explains online ENT care. Different subject, same principle: make the rules visible and write them down.
What this small team took away
None of the fixes were clever. They were authentication, a cache, a list, an owner, a limit and a log setting. The Atlas is a reference rather than a manual, and its entry freely admits that ContainerOps is a loose label with no standard behind it. That honesty is part of why it is useful. It does not sell a framework; it describes habits and measures, and lets a small team choose which ones to start with.
If your own builds ever go red on a Monday for no obvious reason, check the error text before the configuration. A pull limit looks exactly like a broken build until you read the message.