Most organisations discover they have a governance problem the same way.
Something goes wrong. An automated workflow sends the wrong response to the wrong customer. An AI agent escalates an incident that didn’t need escalating. A decision gets made at speed that nobody intended to delegate.
The immediate response is to add a control. An approval gate. A monitoring dashboard. A policy document. Someone is asked to review the logs.
This feels like governance. It is not.
What those organisations have is a running system that was never designed to be governed, with a set of controls sitting on top of it that have no real power over what the system does. The architecture underneath is still acting. The controls are watching.
That is the situation most organisations are actually in right now. Not designing AI governance from scratch. Trying to add it to something already running.
And the answer to how you do that is: you can’t. Not fully. Not without going back to the architecture.
Why controls without architecture don’t hold
There is a version of governance that feels sufficient until it isn’t.
You add logging. You set up alerts. You write a policy that says the system should only act within certain boundaries. You create an approval process for edge cases. On paper, the system is governed.
Then something happens at 2am on a Saturday. The system acts outside the boundaries the policy described. The alert fires. Four minutes later. The approval process requires someone to be awake and watching. Nobody is. By the time a human can intervene, the system has committed three downstream actions that can’t be reversed.
The policy was real. The monitoring was real. The governance wasn’t.
Governance isn’t a set of rules that describe how a system should behave. It’s the ability to enforce those rules at runtime: to intervene when something goes wrong, to stop the system before damage compounds, to understand after the fact exactly what happened and why.
If the architecture doesn’t support those things, the policy is just a document.
Four properties that have to be in the architecture
When I review systems where governance actually holds under pressure, four properties are always present by design. Not added later, not approximated with tooling, but built into how the system is structured.
Policy boundaries the system enforces, not describes. There is a difference between a policy that says “only act on orders under £500” and an architecture where the system is structurally incapable of acting on orders over £500. Most organisations have the former. The system knows the rule. It can also break it. Real policy boundaries mean the rule is enforced by the design of the system around the AI, not by the AI’s judgement.
Escalation paths that exist before they’re needed. Escalation is not a response to incidents. It’s a designed path the system follows when it reaches the edge of its authority. That path has to be explicit: who gets notified, how quickly, with what information and with what ability to act. Systems without designed escalation don’t escalate. They either proceed or fail. Neither is governance.
Intervention that is real, not theoretical. Most organisations can describe how they would stop their AI system. Far fewer have actually tried. Intervention has to be tested: a genuine ability to pause, roll back or redirect the system in the middle of a live workflow. If the intervention path has never been exercised, it probably doesn’t work the way anyone thinks it does.
Auditability that answers the question that matters. Logging is not auditability. Logs tell you what happened. Auditability means being able to answer: what did the system decide, what information did it have at the time, what policy was it operating under and who was responsible for that policy? If you can’t answer those four questions about any action the system took, you cannot govern it. You can only investigate it afterwards.
Going back to a running system
This is where most governance conversations stall. The architecture is live. The system is in production. Nobody can stop to rebuild from scratch.
Not everything needs to be rebuilt. But some things do, and the mistake is patching the things that need to be rebuilt.
The way to approach a running system is to ask, for each of the four properties: do we have this by design, or do we have something that approximates it?
Approximations are fine as temporary positions. A person checking logs every morning approximates auditability. An on-call engineer who can manually stop the system approximates intervention. These things keep the system safe enough while the architecture catches up.
The problem is that approximations break when the system gets faster, handles more volume or gains more autonomy. The engineer can’t check logs quickly enough. The on-call person is asleep. The manual stop can’t operate at the speed the system is running. What held at low volume fails silently at scale.
So the question for any running system isn’t “are we governed?” It’s “which of these properties are we approximating, and what happens when the approximation breaks?” That question tells you exactly where to rebuild and in what order.
What rebuilding actually requires
Redesigning for governance doesn’t mean stopping everything and starting again. It means making a small number of decisions that have been avoided.
The most important is usually boundaries. Not policy documents about boundaries: actual enforcement. Decide what the system is structurally incapable of doing, and make it incapable by design. This is almost always a smaller set of constraints than it initially seems. Most systems can operate within clear limits. The work is deciding what those limits are, not building them.
The second is ownership. Every automated action needs a named owner. Not of the service that ran it, but of the decision it made. That owner decides what acceptable behaviour looks like, reviews escalations and decides when the system needs to change. Without named ownership, governance is everybody’s responsibility, which means it is nobody’s.
The third is the stop mechanism. Before the system gains any more autonomy, test whether you can stop it. Not in theory. Actually pause it, mid-workflow, and observe what happens downstream. If you haven’t done this, you don’t know whether your intervention path works.
These three decisions don’t require rebuilding the whole system. They require rebuilding the parts that governance depends on.
Governance is a property, not a layer
Most organisations treat governance as something to add to a system that is otherwise complete. That framing is why it fails.
Governance isn’t a feature. It’s a set of properties the architecture either has or doesn’t. You can describe policies, build dashboards and assign responsibility all day. If the system can act outside its boundaries, if escalation has no path, if intervention has never been tested and if auditability stops at the logs, the system is not governed.
What you have is a system acting on your behalf, with rules it may or may not follow, watched by people who may or may not be able to stop it.
Going back to the architecture is not an admission of failure. It is the only way to make governance real.
Leave a Reply