{"id":2410,"date":"2026-11-15T18:10:00","date_gmt":"2026-11-15T18:10:00","guid":{"rendered":"https:\/\/askdoctorlive.com\/blog\/?p=2410"},"modified":"2026-11-17T14:10:00","modified_gmt":"2026-11-17T14:10:00","slug":"incidentops-step-by-step-online-service","status":"publish","type":"post","link":"https:\/\/askdoctorlive.com\/blog\/incidentops-step-by-step-online-service\/","title":{"rendered":"IncidentOps Step by Step: Running an Online Booking Service Outage"},"content":{"rendered":"<p>It is a quiet Tuesday evening when the small team behind an online consultation-booking service gets its first sign of trouble. The page that lists available time slots loads forever. A support colleague notices three confused messages from users in ten minutes. Nobody has declared anything yet, and the on-call engineer is cooking dinner. What happens in the next hour will depend less on technical brilliance than on whether the team has a simple routine for unplanned disruption (a routine that falls under the heading IncidentOps).<\/p>\n<p>That routine is what technology teams call IncidentOps, and this guide walks through it step by step, using the booking-service story as a thread. Everything here is about the administration and technology of running a service: who does what, who says what, and when. It is not about medicine in any way, and nothing in it is health guidance. Its practices are drawn from the Ops Atlas, a learner-oriented catalogue of operations subjects, which admits that the name is casual and mostly overlaps with everyday incident management.<\/p>\n<h2>Step 1: Notice, and decide quickly whether it is an incident<\/h2>\n<p>The first job is to stop wondering. Written criteria remove the guesswork. The Atlas points to examples from large technology firms: the problem needs another team, outsiders can see it, or it has not been solved after about an hour. In our story, users can see the failure, so the on-call engineer should declare at once, even though it might turn out to be a five-minute glitch. If it is, the cost is a few minutes of attention. If it is not, the team has saved a lot of time.<\/p>\n<p>A tip: make declaring easy. A single message in the team chat saying &#8220;incident declared: slot list not loading&#8221; is enough. Nobody should feel they need permission or fear that a quick call will count against them.<\/p>\n<h2>Step 2: Hand out the hats<\/h2>\n<p>Within two minutes, the person declaring names who does what. The Atlas calls these roles hats, borrowed for the duration of one incident, and lists four that matter for a small team: a coordinator who directs but does not repair, a hands-on person who changes things, a communicator who writes updates, and a note-taker who records events. In our story the on-call engineer is the hands-on person; the support lead coordinates; a product manager writes updates; and a junior colleague keeps a timeline.<\/p>\n<p>It is worth underlining that the coordinator does not touch the fix. Once the coordinator dives into debugging, the clock and the message queue lose their watcher. If you need to make a call between two options, the coordinator decides, and the hands-on person carries it out.<\/p>\n<h2>Step 3: Size up, stabilise, update, verify<\/h2>\n<p>One published response guide describes a simple loop for the coordinator: size up the situation, stabilise it, update the people waiting, and verify that the stabilising step worked. Then repeat if needed. The loop gives structure to a stressful moment because the next question is always the same. For our story: size up (the slot list depends on a database that is running slowly), stabilise (restart the slow component or move traffic to a standby), update (post a note), verify (reload the page and test a booking).<\/p>\n<h2>Step 4: Tell people early, then keep a rhythm<\/h2>\n<p>The communicator posts the first message within about five minutes of the call, and then sets a timer. Guidance in the Atlas suggests a scoping update shortly after the first, and further updates at least every twenty minutes across the first two hours. The words can be plain: &#8220;Some users cannot see available slots. We are investigating and will post again by 7:20 pm.&#8221; Hosted status tools typically provide four labels, and using them lets users read the situation at a glance: Investigating, Identified, Monitoring, Resolved.<\/p>\n<p>The Atlas also cites a cautionary tale about silence. In April 2022 a large collaboration-software company deleted hundreds of customer sites by mistake during maintenance, and the first widely visible public statement, a social media post, is reported to have come some forty-one hours after the deletion began, although the status page had been touched earlier. Silence feels safe to the team and feels alarming to everyone else.<\/p>\n<p>If you are building your own service stack, our piece on <a href=\"https:\/\/askdoctorlive.com\/blog\/how-to-find-software-that-fits-your-business-and-team\/\">picking software that matches how your business and team actually work<\/a> touches on the same theme of choosing tools around real routines.<\/p>\n<h2>Step 5: Use the shortest path to relief<\/h2>\n<p>The hands-on person now acts. The Atlas stresses that relief often depends on knowing where the problem lives, with no need for a complete diagnosis. Reasonable generic moves include rolling back the last deployment, turning off a new feature, moving traffic to a standby, or limiting load. In our story the engineer notices the slowdown began after an update, rolls it back, and the slot list returns within minutes. The theory of why the update misbehaved can wait for a calmer moment.<\/p>\n<h2>Step 6: Keep the room small and the record clean<\/h2>\n<p>Meanwhile, curious colleagues begin to join the chat, and a senior manager asks how long this will take. Two tidy habits help. First, run a working space for the four hats and a separate updates space that anyone can read but only the communicator writes in. Second, if a senior person starts giving instructions, the coordinator can ask, &#8220;Do you wish to take command?&#8221; Either they accept the hat in so many words, or they step back and leave the coordinator to carry on. The Atlas notes that field research found crowds of hundreds of participants driving core teams into private side chats, which defeats the shared record.<\/p>\n<h2>Step 7: Hand over properly if it runs long<\/h2>\n<p>Suppose the rollback had not worked and the on-call engineer needed to rest. A handover should be spoken and acknowledged: &#8220;You are now the incident coordinator, okay?&#8221; followed by a yes. Silence or a nod over chat is not enough. The Atlas also describes one long case with a handover between continents mid-incident, which shows how routine it can be for a response to outlast a single shift.<\/p>\n<h2>What can go wrong in the middle<\/h2>\n<p>Real evenings seldom follow the script. Three trouble spots are worth rehearsing. First, the alert that should have woken the on-call engineer never arrived, because the alerting service shared a data centre with the failing system. The Atlas advises keeping the paging path and the public status page independent of the thing they report on. Second, the provider the team depends on shows a green dashboard while users are plainly affected. Status pages from providers can lag, so the team should declare on its own criteria and not wait for a vendor to confirm. Third, the first responder is a newcomer who freezes. That is why the Atlas&#8217;s starter plan trains several people as coordinators and runs paging drills, so the first real incident is not the first time anyone has held the hat.<\/p>\n<p>A gentle note on tone: during an incident, short sentences and plain words beat jargon. Users do not need to know which database is slow; they need to know whether they can book, and when to look again.<\/p>\n<h2>Step 8: Close it, then learn from it<\/h2>\n<p>When service is stable and verified, post a closing message that says it is resolved, how long it lasted, and whether anything was lost, such as saved bookings. Then, before logging off, name an owner for the review. Hold the review within a few days. One published process, cited in the Atlas, expects the write-up within three calendar days for its top severity level and within five business days for the next. In the review, look for the conditions that made the failure easy, and avoid ending at a person. Finish with two or three actions, each with one owner and a date.<\/p>\n<p>Teams working with automated machinery may find parallels in <a href=\"https:\/\/askdoctorlive.com\/blog\/robot-fleet-management-essential-concepts-for-robotics-teams\/\">our overview of coordinating fleets of robots across a team<\/a>, where roles and communication matter in just the same way.<\/p>\n<h2>A printable summary of the IncidentOps routine<\/h2>\n<ol>\n<li>Declare on written criteria; do not wait.<\/li>\n<li>Name four hats.<\/li>\n<li>Size up, stabilise, update, verify.<\/li>\n<li>Post a first message fast and keep a steady rhythm.<\/li>\n<li>Prefer the shortest path to relief over finding the cause.<\/li>\n<li>Keep two spaces: working and updates.<\/li>\n<li>Hand over aloud.<\/li>\n<li>Close clearly and review with a few owned actions.<\/li>\n<\/ol>\n<p>The Atlas collects all of these steps, plus a maturity ladder, in <a href=\"https:\/\/www.xopsschool.com\/incidentops\/\">its entry on IncidentOps<\/a>. You can read it start to finish, but even the four-hat summary is enough to start improving your next Tuesday evening. Pick a rehearsal date, try the routine on a made-up outage, and adjust the cards until they fit your team.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A step-by-step IncidentOps walk-through of one evening&#8217;s outage at a small online booking service: declaring, roles, updates, fixing and review.<\/p>\n","protected":false},"author":11,"featured_media":2411,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[731],"tags":[],"class_list":["post-2410","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-healthcare-it"],"_links":{"self":[{"href":"https:\/\/askdoctorlive.com\/blog\/wp-json\/wp\/v2\/posts\/2410","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/askdoctorlive.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/askdoctorlive.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/askdoctorlive.com\/blog\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"https:\/\/askdoctorlive.com\/blog\/wp-json\/wp\/v2\/comments?post=2410"}],"version-history":[{"count":1,"href":"https:\/\/askdoctorlive.com\/blog\/wp-json\/wp\/v2\/posts\/2410\/revisions"}],"predecessor-version":[{"id":2506,"href":"https:\/\/askdoctorlive.com\/blog\/wp-json\/wp\/v2\/posts\/2410\/revisions\/2506"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/askdoctorlive.com\/blog\/wp-json\/wp\/v2\/media\/2411"}],"wp:attachment":[{"href":"https:\/\/askdoctorlive.com\/blog\/wp-json\/wp\/v2\/media?parent=2410"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/askdoctorlive.com\/blog\/wp-json\/wp\/v2\/categories?post=2410"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/askdoctorlive.com\/blog\/wp-json\/wp\/v2\/tags?post=2410"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}