You Can't Know What an Agent Will Do

You know the taxonomy. Known knowns, known unknowns, unknown unknowns. Twenty years of risk decks, and the advice never changes: shrink the third quadrant. Predict harder. Enumerate more. Leave less room for surprise.

You cannot do this with agents.

Non-determinism isn’t a defect waiting for a patch — it’s the reason you delegated in the first place. Same agent, same task, different output, by construction. You can’t even run the same thing twice: every dispatch is a different prompt, and a retry is not a repeat. The input never holds still, so neither does the output. An agent that only did what you predicted is a script, and you already have scripts.

So the real question was never how to eliminate what you can’t foresee. It’s how to run a system where the unforeseeable is welcome — and has nowhere to hide.

Everything that follows rests on one move. You cannot know what an agent will do. But you can make everything around it verifiable — evidence you check, not trust you have to extend:

• what it was — its identity
• what it knew — the structure it operated on
• what it believed — the context it carried
• what it could touch — the boundary around it
• what it did — the actions it took and whether they passed

Five things you can verify, surrounding one thing you can’t. That’s enough, if you build the layer for each. I did. This is the map.

The one that gets you: unknown knowns

There’s a fourth quadrant the risk decks skip. Unknown knowns — the things everyone already knows but nobody has named as a risk.

Here’s one: nine out of ten AI coding tools act as you. Your agent commits with your identity, pushes with your token, calls APIs with your credentials. Everyone knows this. Nobody had filed it as a risk until legitimate agent traffic, running under one human identity, looked enough like an attack that the human had to answer for it.

I know, because that particular unknown known made me somebody’s unknown unknown — a category a security classifier didn’t have a slot for, until my agent traffic tripped it and my account was gone. The classifier wasn’t wrong given what it could see. It just couldn’t see the thing that mattered: a machine acting on my behalf is not me.

That’s the whole project. Everything below is one way of making a machine’s behavior legible before it becomes someone’s incident.

What it was — identity

An agent’s identity should be its own. Not your Personal Access Token. Not a bearer credential that, once stolen, makes the attacker indistinguishable from you.

signet replaces steal-and-use bearer tokens with proof-of-possession: ephemeral certificates (Ed25519 today, ML-DSA-44 post-quantum ready), five-minute lifetimes, per-scope grants. A stolen certificate is useless without the key; a stolen key is worth five minutes, at most. notme.bot applies that to agents, and APAS — the Agent Provenance Attestation Standard — specifies how to represent which specific running execution did a thing. Not “an agent.” This dispatch, bearing this cert, at this moment.

You can watch the industry arrive at half of this in real time. At Ship 2026, Vercel’s Passport put deployed agents behind the same OIDC gate as employees — each agent gets a scoped identity and short-lived, per-task credentials instead of a shared service account. Real progress, and proof the thesis is now consensus: agents need verifiable identity, not standing access. The split is the trust model. Passport’s identity is issued — minted by a central authority and handed to the agent as a signed bearer token, trusted because a gateway vouches for it. signet’s is held — a certificate the agent possesses and cannot export, trusted because it can prove possession of a key. Centralized-and-remote versus dispatch-bound; presentation versus possession. And a bearer token is exactly the thing that, once copied, makes the copy indistinguishable from the original — the failure mode this section exists to close.

Why “dispatch” and not “workload”? Because SPIFFE’s unit of identity is the workload — the program, assumed to be the same actor across runs. Agents break that assumption. The unit of identity has to bind to the execution, because the execution is the only thing that’s real. Same insight as the top of this page, one layer down.

What it knew — structure

An agent given a codebase as raw bytes will hallucinate imports, corrupt files, and burn its context window cat-ing 2,000-line files to find one function.

Your IDE solved this for humans twenty years ago — it doesn’t grep, it asks the language server. mache solves it for agents: it turns your codebase into a navigable graph — functions, types, callers, callees — served over MCP or mounted as a filesystem where ls callers/ literally works. Either way, find the callers is a graph traversal instead of a regex search, and writes are validated against the AST before they land. The agent doesn’t read the 2,000-line file to find one function, because in mache’s projection the file isn’t the unit — the function is. What it knew is no longer a guess — it’s a structure you can inspect. (The full argument.)

What it believed — context

This is the newest wall, and the one almost nothing else addresses yet.

When an agent resumes work — a retry, a handoff between pipeline phases, a warm start — it carries context: a plan, a set of things it thinks it has already read, a belief about the state of the world. If that belief is wrong, every action downstream is wrong, and the error compounds exactly the way unconstrained agents compound errors — except across time instead of across a swarm.

So context gets the same treatment as everything else: content-addressed, so it’s verifiable; hash-chained between phases, so it’s tamper-evident; and a wrong belief is traceable to the dispatch that introduced it. It’s git-bisect for cognition — provenance for ideas, not just code. The handoff chain is already tamper-evident — each phase hash-chains to the last — and the content-addressed envelope is landing in rosary now as warm-resume: an agent restores references it can verify by hash and pulls the full material only when it needs it. Belief that’s been proven still-true, or re-derived. Faith in memory gets replaced with a hash you can check. (The trust primitive underneath.)

What it could touch — the boundary

An agent with your shell has your SSH keys. An agent holding ten services’ credentials is one compromised tool away from spending all of them.

cloister cuts both, from one declared, default-deny policy enforced on two planes at once. On the tool plane, every tool runs in its own isolate with its own scoped credentials, reached over a service-binding channel it can’t forge — a compromised tool can’t touch its neighbors’ secrets, and the master signing key lives in a sign-only helper it never sees. On the kernel plane, the agent harness itself runs sandboxed — Seatbelt on macOS, Landlock on Linux — filesystem allow-listed, network blocked except the credential proxy, ~/.ssh explicitly denied. Not denied on paper: verified denied, EPERM, end to end. The model key never enters the agent’s environment; cloister injects it at the proxy and receipts every call. Next on the same seam: a microVM whose only filesystem is a host-side policy mediator, every read and write authorized per-operation (ADR-0044). What it could touch stops being a promise the agent makes and becomes a fact the kernel enforces. (The credential story.)

What it did — actions

An agent’s work, once committed, is indistinguishable from a human’s. That’s a supply-chain attack waiting to happen: inject malicious code through a compromised agent pipeline and nothing in git blame will ever know.

rosary is the orchestrator, and it accepts work only against constraints — machine-decidable predicates, not opinions. The diff compiles or it doesn’t. Tests pass or they don’t. Three files changed or eleven. Verification is tiered, and the first failure short-circuits. The agent’s opinion about whether its work is good is irrelevant; the predicates are the evaluation. Constraints compose where opinions conflict, which is why this scales to agents and “write clean code” never could. (The methodology.) And the work carries its own proof: every phase handoff is wrapped in a DSSE/in-toto envelope (APAS again), observations carry the authoring agent’s signet cert, and every state-changing call in the cluster leaves a hash-chained, signed receipt — so “what it did” is not a log you trust, it’s a signature chain you verify.

One dispatch, end to end

The layers are easier to trust when you watch a single unit of work move through all five.

rosary dispatches an agent to fix a failing test. Before it runs, signet issues it a certificate scoped to this dispatch — a five-minute lifetime, permission to touch this repo and nothing else. That’s what it was.

It reads the codebase through mache, not as raw bytes: the failing function’s source, the context of imports and types visible to that scope, and ls callers/ to see what a change would break. That’s what it knew — a structure, not a guess.

It’s a warm start — a retry of an earlier attempt — so it restores its prior context by hash: the plan it held, the files it believed it had already read. The chain verifies, so the belief stands; had a hash not matched, the stale belief would trace back to the dispatch that introduced it. That’s what it believed.

It runs inside cloister: shell sandboxed, ~/.ssh denied at the kernel, network blocked except the proxy that injects a scoped token for the one API it may call. A compromised step can’t spend its neighbors’ secrets. That’s what it could touch.

It writes the fix. rosary never asks whether the agent thinks the fix is good — it checks predicates: compiles, tests pass, diff under the limit. Each phase leaves a signed, hash-chained receipt carrying the dispatch’s cert. That’s what it did — a signature chain, not a log.

One task, five independent checks, each attestable on its own and composable into a single account of the run you can replay afterward.

None of that settles whether the fix was the right fix — the maintainable one, the one that actually met the goal. What’s cryptographic is authenticity and integrity: this cert signed this assertion, this chain wasn’t altered. Completeness of context, adequacy of the policy, whether the verifier checked the right property, whether the kernel and the model provider underneath were themselves trustworthy — those rest on threat models and trusted components, not signatures. The map makes an agent’s run legible, not correct. Legible is the part that was missing.

Why it’s a seal, not a stack

Why one system, and not five separate projects?

Surprise flows to the layer you didn’t instrument. Perfect identity with unverified context: the unknown moves into what the agent believed. Airtight verification on top of shared credentials: the unknown moves into who actually ran. You don’t shrink the unforeseeable by hardening one wall — you just relocate it to the wall you skipped. It’s weakest-link logic, applied to epistemics instead of security.

That’s why the layers only work composed. Identity without a boundary is a well-labeled process that can still read your keys. A boundary without verification is a safe box producing unchecked output. Constraints without provenance are predicates anyone can forge the results of. Each piece closes the escape hatch the others would otherwise leave open. Together they don’t reduce what an agent might surprise you with — they make sure that whatever it does, you can name it afterward. Down to the dispatch.

The through-line

Delegation has always demanded one of two things: trust, or verification. For most of software history, verifying the work cost more than doing it yourself, so we defaulted to trust — code review, reputation, “I know this person.” Then agents made verification cheaper than supervision, and the whole equation flipped.

This is the discipline of spending that difference. Not trusting the agent — checking it, at every layer where trust used to sit. Identity you can prove. Structure you can inspect. Belief you can trace. Boundaries the kernel enforces. Actions the math attests.

You still can’t know what an agent will do.

But you can make the conditions and outcomes of what it does independently verifiable — down to the dispatch. What you get is evidence in place of certainty, and something you can check in place of trust. Start anywhere on the map — each piece stands alone, and using one already leaves you better off, but the guarantee is the whole seal.

Open source, all of it: github.com/agentic-research. Built with a Principal-Investigator-plus-AI-lab methodology — I provide the discoveries, strategic direction, and critical analysis; I collaborate with AI to synthesize and articulate. The commit history is public, co-authorship and all. That’s rather the point.