A Plant's SCADA System Can Get Every Alarm Individually Right and Still Fail at the One Thing Alarms Are For

Walk through how most ETP or STP SCADA systems acquire their alarms, and the process looks entirely reasonable at every single step. A new sensor goes in, its alarm gets configured against a sensible setpoint, often the vendor's own commissioning default, and that one alarm is correctly set up on its own terms. Repeat that correctly-made decision across years of plant expansion, new instrumentation, added compliance monitoring, and a plant ends up with a SCADA system where every individual alarm is configured right and the system as a whole has quietly stopped doing the one job alarms exist for.

The point-by-point number and the aggregate number are different questions

Checking whether an alarm is correctly configured is a point-by-point question: does this specific sensor have a sensible threshold, does it trigger when the condition it's meant to catch actually occurs. That question gets asked, reasonably, every time a new instrument is commissioned, and the answer is usually yes. The question that determines whether the alarm management system actually works is a completely different one: across everything the operator is watching, how many alarms arrive per hour, and is that rate anywhere near what one person can actually absorb, understand, and act on. Nobody asks that second question when a single new sensor gets commissioned, because a single new alarm never looks like the thing that breaks an operator's ability to cope. It is the accumulation across years of individually reasonable decisions that does it, and nothing in the point-by-point commissioning process was ever built to catch an accumulation.

Why each new alarm looks harmless and the sum doesn't

A commissioning engineer adding one more alarm to the system is answering a narrow, correct question: is this threshold right for this instrument. They are not, and have no particular reason to be, asking how many alarms per shift the operator watching this screen is now going to see once this one joins everything already configured. Multiply that narrow, correct decision across every instrument a plant has added since its original SCADA build out, DO probes, additional flow meters, new compliance analysers, expanded MCC monitoring, and the operator's total alarm rate can climb steadily for years without a single decision along the way looking wrong in isolation.

This is the same shape of mistake we described in the piece on arc flash study staleness: a system that was correctly specified at every point in time it was checked can still have quietly drifted into a state nobody individually chose, because nothing in the normal course of plant operation ever asks the aggregate question. There, the drift was in fault current and clearing time. Here, it is in alarm rate, but the structure of the failure is identical: a series of locally correct decisions compounding into a system-level problem that no single decision was responsible for and no routine check was ever built to catch.

What an overloaded alarm system actually does

Industry guidance on alarm management (EEMUA 191 and ISA-18.2 are the standards most commonly cited, though published secondary sources disagree noticeably on the exact numeric thresholds each one recommends, so treat any specific figure quoted online with some caution rather than as settled fact) converges on one point regardless of the precise number: there is a rate beyond which a human operator cannot meaningfully process alarms, and plants running standard, uncurated SCADA configurations routinely exceed it. Past that point, an operator does not stop noticing alarms. They start acknowledging them faster than they read them, a coping response to a volume that was never designed to be processed one at a time, which is a completely rational adaptation to an irrational alarm load and also exactly the condition under which a genuinely critical alarm gets acknowledged and dismissed along with everything else, a phenomenon the industry's own term, alarm fatigue, describes precisely.

Picture a plant whose SCADA system started, at commissioning, with a manageable set of alarms an operator could reasonably track through a shift. Over several years the plant added a tertiary polishing stage, an online compliance analyser bank, and expanded instrumentation on an upgraded dosing system, each addition commissioned correctly, each new alarm individually sensible. Nobody at any point re-audited the operator's total alarm burden, because no single addition was large enough to prompt that question, the same way nobody at a growing plant tends to ask whether a flow meter installed years ago still sits in the hydraulic condition it was specified for, until something downstream forces the question. By the time a real upset occurs at the SCADA-overload plant, the operator's screen is already showing a background rate of alarms from sensors that chatter near their setpoints under normal operating variation, nuisance signals that were each individually "correct" when configured, and the one alarm that actually matters arrives embedded in a flood indistinguishable, in that moment, from everything else on the screen.

None of this requires a plant to have done anything wrong. It is, in a real sense, the predictable outcome of doing everything right at every individual decision point while never once stepping back to ask what all those correct decisions add up to in aggregate, which is precisely why the problem is so easy to miss in a routine audit built around checking individual points rather than the system's total output.

Auditing the number nobody was checking

The fix is not re-checking each alarm's configured threshold again, which only re-answers the question that was already being answered correctly all along. It is a periodic alarm rationalisation review that asks the aggregate question directly: pulling alarm history over a representative period, counting actual alarm rate per operator per shift, and identifying which alarms fire often enough to be background noise rather than genuine, actionable signals. Alarms that chatter near a threshold under normal variation, or that duplicate information another alarm already provides, are typically the largest contributors once a plant actually measures its own alarm load rather than assuming the point-by-point commissioning process kept it under control. This is a data exercise a plant can run against its own historian without new instrumentation or new software, and it answers a question that nothing in the routine commissioning process was ever designed to ask.

If your plant's SCADA system has added instrumentation steadily over the years without anyone auditing the operator's total alarm rate against it, that gap is worth closing with Spans' instrumentation and controls team before the next real upset arrives buried in noise nobody budgeted for.