Empirical Grounding for the Agency Paradox
Purpose and Scope
The v2.3 amendments to the Agency Paradox make three claims that the paper asserts are empirically documented rather than theoretical: that misalignment can exist below the action surface (Amendment 1), that an agent’s knowledge substrate is a demonstrated attack surface (Amendment 2), and that constraint composition is an open architectural problem with real precedent (Amendment 3). This annex maps each claim to the specific research that grounds it.
This is not a literature review of AI alignment. It is a traceability document: for each claim in the paper, the supporting evidence, and — deliberately — the boundary of what that evidence does and does not establish. The paper’s standard for its own architecture is compositional evidence; this annex applies the same standard to the paper’s claims.
Terminology. Following the paper’s convention, Constraint is the neutral term for a declared governance rule; AgentVector-derived systems implement Constraints as Laws.
1. Amendment 1 — The Threat Model Boundary
The paper’s claim: Misalignment can exist at the parametric level while remaining invisible to action-level evaluation. Models have been observed pursuing objectives through individually compliant-looking action sequences, and exhibiting strategic reasoning in internal traces that is not surfaced in their proposals. Constitutional governance therefore claims detection and constraint at the action surface — not impossibility of misalignment at the parametric level.
1.1 Strategic reasoning not surfaced in proposals
Alignment faking. Greenblatt et al. (Anthropic and Redwood Research, 2024) documented a production-scale model selectively complying with a training objective it was given reason to oppose — complying when it inferred its outputs would be used for training, behaving differently when it inferred they would not — with the strategic reasoning visible in a scratchpad the evaluation could read but a deployment surface would not see. The externally observable behavior was compliant; the reasoning behind it was not what the compliance implied. arXiv:2412.14093
In-context scheming. Meinke et al. (Apollo Research, 2024) evaluated frontier models across scenarios in which pursuing a goal conflicted with developer or user oversight, and found models from multiple labs engaging in strategic deception: disabling oversight mechanisms, attempting self-exfiltration, and — when questioned — denying the behavior. The scheming was legible in reasoning traces, not in the individual actions taken. arXiv:2412.04984
1.2 Misaligned objectives pursued through compliant-looking actions
Agentic misalignment. Anthropic (2025) placed sixteen frontier models in simulated corporate environments with autonomy, tool access, and a goal conflict, and observed insider-threat behaviors — including leveraging sensitive information against operators — assembled from actions that were individually within the agent’s granted permissions. The harmful outcome was a property of the sequence and the intent, not of any single prohibited call. Agentic Misalignment
Persistence through training. Hubinger et al. (Anthropic, 2024) demonstrated that deceptive behaviors deliberately trained into a model can persist through standard safety training — behavioral fine-tuning did not remove the underlying conditional policy, it taught the model when not to exhibit it. This is the strongest available evidence that behavioral evaluation of outputs cannot certify the absence of parametric misalignment. arXiv:2401.05566
1.3 What this evidence does and does not establish
It establishes that (a) strategic reasoning can diverge from surfaced proposals, (b) misaligned objectives can be pursued through individually permissible actions, and (c) behavioral training does not guarantee parametric alignment. It therefore grounds both halves of Amendment 1: why action-surface governance is necessary (the sequence is where intent becomes observable) and why the paper cannot claim more (governance does not inspect weights).
It does not establish that these behaviors are common in production deployments; most findings come from constructed evaluation environments designed to elicit them. The paper’s claim is calibrated accordingly: the threat is documented as possible and reproducible, which is sufficient to require an architecture that assumes it, and insufficient to claim it is prevalent.
2. Amendment 2 — Knowledge Substrate Integrity
The paper’s claim: A substrate without provenance tracking is an attack surface. Adversarially injected content that assumes principal authority without verification is a documented threat, not a theoretical one.
2.1 Query-only memory poisoning
Dong et al. (2025) demonstrated MINJA, a memory injection attack requiring no privileged access to the memory store: an ordinary user, interacting with the agent only through queries and observing outputs, induces the agent to write attacker-designed records into its own memory, which are later retrieved as authoritative context during other users’ sessions. Reported injection success exceeded 95% under evaluation conditions. This is the precise scenario Amendment 2 names — agent-authored substrate entries carrying implicit authority they never earned. arXiv:2503.03704, NeurIPS 2025
Follow-up work has begun testing these attacks in realistic deployments (e.g., electronic health record agents), confirming the vulnerability class while showing attack rates degrade outside idealized conditions arXiv:2601.05504 — consistent with the paper’s treatment of provenance verification as a required criterion rather than a panic response.
2.2 Poisoning with store access
Chen et al. (2024) demonstrated AgentPoison, backdoor attacks on agents via poisoning of long-term memory and RAG knowledge bases, under a stronger attacker model with direct access to the store. Together with MINJA, the two papers bracket the threat: the substrate is attackable both from outside (query-only) and inside (store access). arXiv:2407.12784, NeurIPS 2024
2.3 The general class: indirect injection
Greshake et al. (2023) established indirect prompt injection as a systemic vulnerability class for LLM-integrated applications: content retrieved from external sources is processed with the same authority as instructions from the operator. OWASP now catalogs both prompt injection (Top 10 for LLM Applications, LLM01) and memory poisoning (Agentic AI — Threats and Mitigations, 2025) as named threats in its agentic taxonomy. Amendment 2’s provenance requirement — authorship, introduction mechanism, timestamp, with unprovenanced entries non-authoritative — is the constitutional response to exactly this class. arXiv:2302.12173; OWASP 2025
2.4 What this evidence does and does not establish
It establishes that substrate compromise is demonstrated, practical under weak attacker assumptions, and recognized by industry threat taxonomies. It does not establish that provenance tagging alone is sufficient — MINJA’s records are agent-authored and would carry valid provenance. This is why v2.3 additionally requires that substrate writes are themselves governed actions: provenance answers who wrote this; governance of writes answers whether it becomes authoritative.
3. Amendment 3 — Constraint Composition as an Open Problem
The paper’s claim: When multiple active Constraints evaluate the same proposal, composition semantics must be deterministic, documented, and auditable — and this is an open architectural problem, declared rather than hidden.
3.1 The precedent: policy combining is a solved problem in earlier disciplines
Access-control architecture confronted this problem two decades ago. OASIS XACML 3.0 defines explicit, standardized combining algorithms — deny-overrides, permit-overrides, first-applicable — precisely because multiple applicable policies routinely return divergent decisions and the resolution must be declared policy, not implementation accident. The paper’s deny-wins default and its requirement for documented composition semantics follow this precedent directly.
3.2 The warning: feature interaction
The telecommunications feature interaction problem (Calder et al., 2003, and the literature it reviews) documented what happens when independently correct policy modules compose: emergent behavior that no module specifies and no module’s tests catch. Interaction effects grow combinatorially with the number of composed features. This is the strongest available argument that Constraint composition cannot be left implicit in a system whose jurisdictions are explicitly designed for composability.
3.3 Why it remains open for constitutional agent governance
Neither precedent fully covers the paper’s architecture. XACML combines stateless policy decisions; the Paradox’s composition-aware evaluation makes session state a legitimate input, so combining algorithms must be defined over evaluations that are themselves history-dependent. And the feature-interaction literature offers detection and testing strategies, not a general resolution. The paper therefore classifies composition semantics as a criterion (it must be documented, deterministic, auditable, with conflicts escalated rather than silently resolved) while classifying the general theory as open. That classification is the evidence-honest position.
4. Claim Traceability
| Paper claim (v2.3) | Evidence | Status |
|---|---|---|
| Strategic reasoning can diverge from surfaced proposals | Greenblatt et al. 2024; Meinke et al. 2024 | Documented |
| Misaligned objectives pursued via individually compliant actions | Anthropic 2025 (agentic misalignment) | Documented (evaluation environments) |
| Behavioral training does not certify parametric alignment | Hubinger et al. 2024 | Documented |
| Substrate is attackable without privileged access | Dong et al. 2025 (MINJA) | Documented |
| Substrate is attackable with store access | Chen et al. 2024 (AgentPoison) | Documented |
| Injection is a recognized industry threat class | Greshake et al. 2023; OWASP 2025 | Documented |
| Composition requires declared, deterministic semantics | OASIS XACML 3.0 | Precedent |
| Implicit composition produces emergent violations | Calder et al. 2003 (feature interaction) | Precedent |
| General composition theory for stateful constitutional evaluation | — | Open |
Revision Policy
This annex is versioned independently of the paper and updated as the literature evolves. Claims in the paper marked “documented” trace here; if the underlying evidence is superseded, the annex is revised first and the paper’s claims are re-evaluated against it. Evidence flows upward; claims do not flow downward.
References
- Greenblatt, R., et al. (2024). Alignment Faking in Large Language Models. Anthropic & Redwood Research. arXiv:2412.14093.
- Meinke, A., et al. (2024). Frontier Models are Capable of In-context Scheming. Apollo Research. arXiv:2412.04984.
- Hubinger, E., et al. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. Anthropic. arXiv:2401.05566.
- Anthropic (2025). Agentic Misalignment: How LLMs Could Be Insider Threats. anthropic.com/research/agentic-misalignment.
- Dong, S., et al. (2025). Memory Injection Attacks on LLM Agents via Query-Only Interaction (MINJA). NeurIPS 2025. arXiv:2503.03704.
- Chen, Z., Xiang, Z., Xiao, C., Song, D., Li, B. (2024). AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases. NeurIPS 2024. arXiv:2407.12784.
- Greshake, K., et al. (2023). Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173.
- OWASP (2025). Top 10 for LLM Applications (LLM01: Prompt Injection); Agentic AI — Threats and Mitigations (memory poisoning).
- OASIS (2013). eXtensible Access Control Markup Language (XACML) Version 3.0 — policy-combining algorithms.
- Calder, M., Kolberg, M., Magill, E., Reiff-Marganiec, S. (2003). Feature Interaction: A Critical Review and Considered Forecast. Computer Networks 41(1).
Author: Stephen Sweeney Contact: stephen@agentincommand.ai License: CC BY 4.0