When an agent can delete production.

An agent used a locally stored, broadly scoped token to delete a production volume through a legacy API. The infrastructure provider later reported that it recovered the database with all data and extended its delayed-deletion protection to that API. Deletion, disruption, and permanent loss are different outcomes; this case establishes the first two, not the third.

The mechanism matters more than the agent's explanation afterward: it could reach a destructive operation using credentials broader than its task required. Instructions about what it should do were not the same as limits on what that API would accept.

A signed record would help a reviewer inspect a recorded decision. It would not itself prevent this deletion or recover data. Scoped credentials and reversible operations are not in the published AGA packages; deployers configure those controls and test their own backups.

The provider's account distinguishes volume-level backups from offsite disaster backups. It also describes narrower token scopes that already existed. The lesson is to apply the available controls to every execution path, including the APIs agents can reach directly.

This is not a one-off

The pattern isn't new. Coding agents have deleted production databases during code freezes, wiped production setups including their snapshots, and kept deleting a user's messages after being told repeatedly to stop. Outages have been traced back to automated coding agents. The list keeps getting longer.

A recurring risk is a mismatch between task-level instructions and the authority available through credentials. Where production separation exists only by convention, an agent can cross that boundary without defeating any access control. Whether that risk exists must be checked in the actual deployment, not inferred from the model or industry.

The usual defense of this pattern goes: deployers should have had better backups, scoped credentials more tightly, run agents in sandboxed environments. All of that is true in detail and wrong in implication. It puts the burden on every team using AI agents to invent the safety architecture from scratch before they ship anything. That doesn't scale when every industry is deploying agents at once. The safety architecture has to come as a default. Otherwise each deployer rebuilds it from memory, and each one breaks in a new way, depending on what they happened to forget.

The architectural question

One way to reduce failures like this is behavioral safety: better training, better prompts, better reinforcement learning from human feedback. The other is architectural safety. Behavioral safety tries to make agents less likely to take unauthorized actions. Architectural safety tries to make those actions impossible to execute, whatever the agent decides.

The deletion illustrates why instructions alone are not an access boundary. On this path, the credentials and API permitted a destructive operation. Narrower permissions, approval requirements, and recovery controls address different parts of that risk; a signed decision record does not replace them.

Four mechanisms, each addressing a specific failure in the chain. For each, here is what the published AGA packages provide.

A policy the runtime applies. The deployer writes down what the agent is allowed to do, where, and against which resources, and something outside the agent applies it before any action reaches the infrastructure. A decision to delete a volume in response to a credential mismatch would not be in that policy. In the published aga-proxy, the policy is a JSON file (tool allowlist, rate limits, path and substring rules; known issue 10 on /security says what they check), and every receipt carries the signed SHA-256 of the policy's canonical JSON; it governs the tool calls routed through the proxy.

A boundary that holds the credentials. Destructive operations go through a separate process that holds the credentials. The agent doesn't have them. It has to request the operation, and the boundary checks it against the policy first. This isn't a sandbox, which an agent holding credentials can bypass; the point is that the agent can't do the operation without going through something that can refuse. A token lying in an unrelated file wouldn't matter if it didn't live anywhere the agent could read. aga-proxy is that kind of boundary for MCP tool calls, and in-path it does not forward a call it denies (its default profile, permissive, denies nothing on policy grounds; policy denial needs --profile standard or restrictive, or a --policy file in allowlist or denylist mode); keeping credentials out of the agent's reach is part of how you deploy it.

Credentials scoped to specific subjects and actions. The token in this failure was effectively scoped to anything its user could do. The alternative is tokens bound to one subject (this agent, in this environment, for this task), deny-by-default for everything else. That is a property of your identity and cloud configuration; AGA does not issue or scope credentials.

Evidence kept apart from the data it describes. An investigation should compare the agent's account with provider records, access logs and recovery evidence. A signed decision record can add a check of retained values against an expected key; it does not replace that investigation. The provider reported recovering the database, so this case must not be presented as permanent loss of all backups. Separately retained records can reduce exposure to the same deletion path, depending on storage permissions. aga-proxy keeps its receipt chain in memory: retain exported bundles outside that process if they must survive its termination. The export is available through GET /export on its loopback control port and can be verified offline.

Access restrictions, approval steps, recovery controls and signed records all have established implementations. Cryptographic primitives such as Ed25519 and SHA-256 address record integrity; they do not supply the whole safety architecture. Integration effort and coverage need testing in the actual deployment.

The vendor trap

Model providers, application vendors and deployers control different parts of a system. Access restrictions, approval steps, collection and recovery need to be assigned to the party that can implement them. A deployer may already have useful provider and infrastructure logs. The question is what those records establish and whether they cover the disputed path.

Independence is a property to examine, not something the word “audit” guarantees. Ask who observes an event, who can alter or omit its record, where copies are retained and which key a recipient expects. These questions apply to provider logs, independent collectors and AGA alike.

The regulatory question

A public incident report does not establish the frequency or severity of unreported failures. It can still motivate a bounded review of credentials, destructive API paths and recovery controls in a particular deployment.

Regulators are pointing at the same gap. The EU AI Act's high-risk obligations phase in across 2027 and 2028[1], the NIST AI RMF continues to be updated[2], and CISA's Secure by Design and Secure by Demand guidance asks software buyers to press vendors on these practices during procurement[3]. Each assumes organizations can show how their systems were governed. None of them requires signed records, but each is easier to answer with a record someone else can check.

What comes next

The mechanisms already exist, and shipping them needs no regulatory mandate. The teams that deploy them first will have an easier time answering their customers' and regulators' questions.

The question for anyone running AI agents in production is not will this fail. It is when this fails, will we know what happened, can we show it to whoever asks, and did we build the failure mode out of the architecture before we deployed? Those are governance questions, and they have technical answers.

What is missing is the understanding that safety architecture is a precondition for deployment rather than something bolted on afterward.

References

1. Regulation (EU) 2024/1689 (AI Act), Article 113, as amended by Regulation (EU) 2026/1744 of 8 July 2026. Official Journal of the European Union. eur-lex.europa.eu. The high-risk requirements apply from December 2, 2027, for systems classified as high-risk under Article 6(2) and Annex III, and from August 2, 2028, for those under Article 6(1) and Annex I.
2. “AI Risk Management Framework.” NIST. nist.gov.
3. “Secure by Design.” CISA, including its Secure by Demand guidance. cisa.gov/securebydesign.