On Friday 13 November, a Microsoft outage took out Outlook for a lot of people. For most of them it meant a quiet afternoon and a backlog on Monday. At Phantasialand, a theme park and hotel complex in Brühl, Germany, it meant something else. According to WDR's report on the chlorine incident at Phantasialand, the pool area's automatic dosing system pumped too much chlorine into the water, the monitoring system raised the alarm by email, and nobody received the email. Fifteen guests at the Hotel Ling Bao were hurt. Thirteen initially complained of respiratory irritation, the number rose to fifteen, four were taken to hospital, and one stayed overnight for observation with a moderate injury.
Emergency services tested the water, found the fault in the automatic dosing system, and the pool was drained and refilled. Phantasialand told WDR that because of the disruption, no mail came through, so nobody noticed that the safety system had already sounded the alarm.
That is a horrible story, and it is also the clearest illustration you will ever get of a problem almost every business has and almost nobody has written down.
The outage was the trigger, not the cause
The tempting read is "the cloud failed". That read is wrong and it will lead you to the wrong fix. Microsoft 365 going down for a few hours is a normal, expected, forecastable event. Every hosted service you use will be unavailable at some point this year. The question is never whether the dependency fails. It is what your business does while it is failing.
The commenters on the Hacker News thread about the incident landed on this fast and were fairly brutal about it. Several argued the fault sits with the system design rather than with Microsoft: one wrote that anything capable of causing that kind of harm needs a more resilient and redundant architecture, and another asked why a safety alarm was an email at all, rather than a horn or a flashing light. Treat those as opinions, not findings. But they are the right opinions.
Somewhere in that build, someone made a reasonable-sounding decision. The dosing system can send alerts. Email is how we reach staff. Point the alerts at email. Every step is sensible. The combined result is a safety alarm whose entire delivery path runs through a third party in another country, with no local sound, no light, no second channel, and no rule saying what happens if the pool chemistry is out of range and nobody has acknowledged it.
Your version of this is smaller, and you still have one
You do not run a pool. Fine. You almost certainly still have processes that silently sit downstream of a login you do not control.
If your booking platform is unreachable, do your staff know whether today's bookings exist on paper anywhere? If your point-of-sale provider has an outage at 12:30pm on a Friday, is the answer "we take cash and write it down", or is it fifteen minutes of people looking at each other? If your supplier orders go out by an automated email, how would you know they had stopped going out? If your cool room, freezer, or server room has a temperature alarm, what does that alarm actually do, and who does it reach when the internet connection at the premises is down?
The pattern in the Phantasialand case is worth naming precisely, because it is the one that catches people. The dangerous failure was not that a system stopped working. It is that a system kept working while its warnings went nowhere. Silence looked identical to everything being fine.
The hour that is actually worth spending
This is a whiteboard exercise, not a project. Get the people who run the day-to-day in a room and build a single list with three columns.
- The service. Microsoft 365, Xero, your POS, your booking engine, Shopify, your SMS gateway, your phone system, the site's internet connection.
- What stops. Not the software. The actual business activity. "We cannot take payment." "Orders do not reach the kitchen." "Rostered staff do not get told about shift changes." "The temperature alarm does not reach anyone."
- What we do instead. A decision made now, in writing, by someone who is not under pressure.
Most lines will resolve to "we wait, it is annoying, nothing is harmed". That is a fine answer and writing it down is still useful, because it means you have ruled the line out deliberately rather than by not thinking about it. The value of the exercise is the two or three lines where the honest answer is "actually, something bad happens and we would not know".
Those lines need a fallback that does not share a failure path with the thing that broke. If the alert matters, it should reach a human by more than one route: a local siren or light at the site, an SMS from a different provider, a phone call. And it should be designed so that no alert at all is also a signal. A monitoring system that goes quiet should itself trigger something, because "we have heard nothing" and "everything is fine" must not look the same.
The question to put in your next vendor contract
If you are buying or commissioning any system that controls, monitors, or triggers something physical or time-sensitive, ask the supplier one question in writing before you sign: what happens to this process when your service is unavailable, and how will we know?
A good supplier answers in specifics. The device keeps running on its last known settings. It raises a local alarm on-site independent of the network. It queues and retries, and it tells you when the queue is backing up. A weaker supplier answers with uptime percentages, which describe how often they expect to be fine, not what happens to you when they are not.
This is also how we think about it as a studio. When we scope a build, dependencies on third parties are unavoidable and mostly good: you should not be writing your own payment processor. What is not acceptable is an undocumented dependency sitting underneath something that can hurt someone, lose money quietly, or breach an obligation. If a notification matters, it gets a second path. If a process must not stall silently, the absence of a signal has to raise its own alarm. That costs very little when it is designed in at the start, and it is expensive and awkward to retrofit after an incident.
What to do this week
Write the list. One hour, three columns, the people who actually do the work. Mark the lines where failure is silent rather than obvious, and fix those first. Then test one of them for real: pick a quiet Tuesday, pretend the service is down, and see whether the written answer survives contact with your actual staff.
And if there is any alarm in your business whose only delivery mechanism is email, treat that as the first thing to change. Email is a good way to tell someone something. It is a bad way to be certain they heard it.


