Last year, a Fortune 500 company I was advising deployed what they called an "AI automation pilot." The system could read emails, schedule meetings, draft responses, and — if it judged a decision to be routine — execute financial transactions up to a $10,000 threshold without human review.
It worked brilliantly for three weeks. Then it didn't.
The failure wasn't a hallucination in the traditional sense. The agent made a coherent, logical series of decisions that individually looked reasonable and collectively produced an outcome that no human in the organization would have sanctioned. The agent had been given an objective, tools to achieve it, and a threshold for autonomous action. Nobody had adequately specified what the agent wasn't allowed to do. They had built a capable system. They hadn't built a governed one.
This is the agentic AI governance problem in miniature. And it's playing out at scale across every enterprise that is deploying these systems right now.
Why Existing AI Governance Frameworks Fail for Agentic Systems
The AI governance frameworks that most enterprises have in place today were designed for a different kind of system: supervised models that take an input, produce an output, and return control to a human. Bias audits, explainability requirements, model cards, output monitoring — these are all valuable tools for that category of system. They are largely inadequate for agentic AI.
The fundamental difference is agency over time. A traditional AI system makes one decision per invocation. An agentic system makes dozens or hundreds of decisions per task, each one potentially changing the context for the next. The failure modes that matter are not single-output errors — they are drift across a decision chain, where each individual step looks reasonable but the cumulative trajectory violates implicit constraints the system was never told about.
Consider the dimensions along which agentic systems differ from supervised models:
Multi-step planning. An agentic system that can plan across a time horizon has an implicit model of the future. That model can be wrong in ways that compound — a faulty assumption about step three becomes a constraint on steps four through ten before anyone notices.
Tool use and real-world effects. Agentic systems execute code, call APIs, write to databases, send communications. Their outputs are not tokens on a screen — they are actions with irreversible consequences. The governance framework needs to address not just what the agent says but what it does. This is where hardware security becomes relevant: in production deployments, agentic systems execute on cloud infrastructure where the hardware layer itself represents an attack surface. Side-channel attacks on ML accelerators and the confidential computing infrastructure designed to protect against them are part of the governance-aware deployment checklist for any agentic system handling sensitive data.
Multi-agent delegation. Increasingly, production agentic systems involve hierarchies of agents: an orchestrator that breaks down tasks, specialist subagents that execute them, and other agents that evaluate results. Governance requirements need to propagate through the full delegation chain, not just apply to the orchestrator.
Emergent behavior. The interaction of multiple agents, each behaving according to its own logic, can produce system-level behaviors that none of the individual agents were designed to produce. Standard unit-level model evaluations don't capture this.
I spent the first several years of my AI research career working on systems that could plan and execute — the autonomous agents that emerged in academic research settings before the transformer era made them practically viable at scale. What I saw then, and what I see now with far more capable systems, is the same pattern: capability outruns governance because capability is legible and governance is hard.
The Four Pillars of a Working Agentic AI Governance Framework
Based on my experience building and deploying agentic systems — through my AI ventures, my research at NYU Tandon and IIT Kharagpur, and the enterprise deployments I've advised — I've come to believe that effective agentic AI governance requires four things. Most frameworks I see in the wild address one or two of them partially. Almost none address all four.
1. Constrained Action Space Specification
Before deploying any agentic system, you need an explicit, machine-readable specification of what the agent is allowed to do. Not just what it's designed to do — what it's permitted to do.
This is not the same as a system prompt. A system prompt is instruction; an action space specification is constraint. The agent's underlying model should be unable to execute actions outside the permitted set regardless of what the system prompt says, what the user requests, or what reasoning the agent generates to justify crossing a boundary.
In practice, this means tool definitions need to be accompanied by permission specifications. A web browsing tool should have a domain whitelist and a data-exfiltration prevention spec. A code execution tool should have a sandboxing specification and a resource consumption limit. A database access tool should have a read/write permission model that specifies not just which tables but which operations under which conditions.
The hardest part of this is the specification exercise itself. Most organizations know what they want their agents to do. They haven't fully specified what the agents are not allowed to do. The governance process should force that specification before deployment, not after a failure.
The agentic AI security research I've been following closely makes clear that unconstrained action spaces are the primary attack surface for both adversarial and accidental failure. Prompt injection attacks — where malicious content in a tool result attempts to redirect the agent's behavior — rely on the agent having authority to take actions that a well-specified permission system would have prevented regardless of the injected instruction.
2. Reversibility Classification and Escalation Triggers
Not all agentic actions are equal. Some are easily reversible — drafting a document, querying a database, running an analysis. Others are difficult or impossible to reverse — sending an email, executing a financial transaction, modifying a production database, deploying code. A well-designed governance framework treats these categories differently.
The principle I apply in enterprise deployments is what I call reversibility tiering:
- Tier 1 (Freely reversible): Agent executes autonomously, logs action for audit, no approval required. Includes read operations, draft creation, internal computations.
- Tier 2 (Soft reversible): Agent executes, flags for human review within a defined time window. Includes low-stakes external communications, non-critical data writes. Human can recall within the window.
- Tier 3 (Hard to reverse): Agent requires pre-execution human approval. Includes external communications with financial implications, database modifications affecting live systems, API calls to external services with real-world consequences.
- Tier 4 (Irreversible): Agent cannot execute without explicit board-level or committee approval. Includes financial transactions above threshold, code deployments to production, public-facing content publication.
The tiering specification should be defined by the business, not the engineering team — the people who understand consequence own the tier assignment. The engineering team implements the enforcement mechanism. This separation is important because it prevents the system from being governed by whoever is most technically confident rather than whoever understands the risk.
Escalation triggers are the corollary: conditions that should cause the agent to stop and request human input regardless of whether it "knows" what to do. An agent that encounters a situation outside its training distribution should be able to recognize the uncertainty and escalate rather than proceeding with a low-confidence decision. Building that metacognitive layer — genuine uncertainty awareness rather than confidence theater — is one of the harder engineering problems in agentic AI, and it's one that most current systems handle inadequately.
3. Audit Trail and Explainability for Decision Chains
Governance requires accountability. Accountability requires the ability to reconstruct what happened and why. For agentic systems, this means audit trails that capture not just the outputs but the decision chain — the intermediate reasoning steps, tool calls, intermediate results, and delegation events that led from the initial task to the final action.
This is technically non-trivial. Agentic systems running on LLM backends generate reasoning through processes that are partially opaque even to the model itself. The chain-of-thought is a useful artifact, but it's a post-hoc reconstruction as much as a transparent record of the actual decision process. Current explainability techniques are inadequate for multi-step agentic behavior, and the research community is appropriately focused on this problem — see the recent work on agentic AI security and interpretability for the current state of the art.
What organizations can do in the near term: instrument heavily. Log every tool call with its inputs and outputs. Log every agent-to-agent delegation with the instruction passed and the result returned. Log the full context window at decision points where irreversible actions are taken. Store these logs with tamper-evident integrity guarantees. Most enterprise deployments I've reviewed log far too little — often just the final output — and then find themselves unable to reconstruct failure modes when things go wrong.
The audit infrastructure should be designed for the failure investigation that will eventually happen, not for the nominal operation that you're hoping to achieve.
4. Continuous Red-Teaming and Adversarial Testing
The failure mode I described at the opening of this piece — an agent making a coherent chain of individually reasonable decisions that collectively violated organizational intent — is not caught by standard model evaluation. It requires adversarial testing specifically designed to probe the agent's behavior at the edges of its defined behavior space and in conditions its designers didn't anticipate.
This is what I call behavioral stress testing for agentic systems, and it's significantly harder than the red-teaming practice that has become standard for LLM deployment:
Objective drift testing: Give the agent tasks with conflicting intermediate incentives and observe whether it maintains the primary objective or drifts toward locally optimal but globally wrong solutions.
Constraint boundary testing: Design test scenarios that would benefit the agent (as measured by its reward signal) from violating its specified constraints, and verify that constraints are actually enforced rather than just weakly discouraging violations.
Delegation chain manipulation: In multi-agent systems, test whether a compromised or malfunctioning subagent can induce the orchestrator to take actions it shouldn't. Multi-agent security research has shown this is a significant and underappreciated attack surface.
Prompt injection via tool results: Verify that tool outputs containing adversarial instructions don't redirect agent behavior. This is the agentic equivalent of SQL injection, and it's the most commonly exploited real-world vulnerability in deployed agentic systems.
Environmental shift resilience: Test agent behavior when the environment differs from training conditions in ways that could affect its constraint model — time zone changes, currency conversions, different user permission levels, edge cases in tool APIs.
Red-teaming should be a continuous process, not a pre-deployment gate. Agentic systems change when you update their underlying models, when you add or modify tools, when you change system prompts, and when the environments they operate in change. Any of these changes can invalidate previous safety validation.
What Organizational Governance Structures Need to Change
Technical governance frameworks are necessary but not sufficient. The organizational structures around agentic AI need to change too, and this is where most enterprises are furthest behind.
The "AI safety" function needs to be upstream of "AI deployment." In most organizations, safety review is a gate that happens before deployment and rarely afterward. For agentic systems, the safety function needs to be embedded in ongoing operations — with authority to pause or constrain running systems when new risks are identified.
The people who understand consequences need to own tier assignment. The reversibility tiering framework I described above only works if the tier assignments reflect actual organizational risk tolerance. That requires the CFO to have an opinion about financial transaction thresholds, the General Counsel to have an opinion about communication authority, and the CISO to have an opinion about data access permissions. These conversations need to happen before deployment, not after a failure triggers a retrospective.
Incident response plans need to account for agent-caused incidents. When an agentic system causes a problem, the response is different from when a human causes the same problem. The speed of recovery depends on the audit trail quality I described above. The remediation often requires understanding an emergent multi-step failure that no single decision in the chain would have flagged. Organizations that are deploying agentic systems today should be developing incident playbooks for agent-caused failures now, before those failures occur.
The State of the Art and Where We Need to Go
I'm building agentic systems at Snow Mountain AI and advising on enterprise deployments across financial services, healthcare, and critical infrastructure. The honest assessment of where the field is right now: technical capability is running significantly ahead of governance maturity.
The good news is that the governance framework I've described — constrained action spaces, reversibility tiering, audit trail design, continuous red-teaming — is implementable with current tools and engineering practices. It doesn't require waiting for interpretability research to mature or for regulatory frameworks to be finalized. Organizations that implement it now will be significantly better positioned when regulation does arrive, because they'll have the documentation, audit trails, and testing evidence that regulators will require.
The less good news: most of the agentic AI deployments I'm aware of today have none of these elements in place. They have capable agents and inadequate governance. That gap is going to produce failures — probably several high-profile ones — before the field settles on practices that match the risk profile of the technology.
The $10,000 autonomous transaction threshold that failed for my client last year has become a $100,000 threshold at other organizations, with the same inadequate governance frameworks. The capability has scaled. The governance has not.
This is fixable. But it requires treating governance as a first-class engineering problem rather than a compliance checkbox. The research on agentic AI safety makes clear what the technical risks are. The organizational will to address them before rather than after failure is what's actually missing — and that's a leadership problem, not a technology problem.
The organizations that get this right in the next 18 months will have a significant advantage when the failures at organizations that didn't make the governance investment start making headlines. I'd rather be in the former category. If you're building or deploying agentic systems at scale, I'd urge you to think carefully about which category you're in.