The Global Centre for Risk and Innovation (GCRI)
AI agents can draft analyses, run pipelines and query data at a speed no team can match, which is why their authority must be explicit. An agent that can call tools can call the wrong one, read data it should not, follow instructions planted in a document, or present a draft as reviewed evidence. This page sets out how agent risk develops and spreads, how agents are evaluated before they receive authority, and the controls that let people stop them.
AI agent risk is the risk that an AI system able to call tools and act takes the wrong action, reads data it should not, follows instructions planted in a document or presents a draft as reviewed evidence. Explicit authority records, permissions enforced outside the agent, kill switches, receipts and human review contain it.
The working screen behind this subject: its structure, terms and states. Names are placeholders and values illustrative.
Agents and automation · Project space 12 · Cluster east
| J-86161 | Flood model ensemble | Project owner | 00:37:12 | Running |
| J-42787 | Tile render, zoom 0 to 12 | GIS workspace | 00:02:40 | Running |
| J-76872 | Evidence export | Model steward | 00:00:47 | Done |
| J-55145 | Agent tool call batch | Agent sandbox | 00:01:16 | Policy block |
AI tools, authority and risk controls. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.
Each tool an agent can call adds a path from a model error to a real effect: a query, a write, a message, a run. Risk grows with the number of tools and the privileges behind them, not with the model alone.
Text retrieved from a web page, a document or a dataset can carry instructions the agent then follows. Prompt injection turns ordinary content into a control channel, so tool permissions must never depend on the agent recognising it.
Language models produce confident, well-formatted output that can be wrong, including invented citations and figures. Once an unlabelled draft enters a report, it borrows the authority of the report and the people who signed it.
One agent's output becomes another's input, or triggers a pipeline, with no person in between. An early error is copied, transformed and amplified along the chain, and the receipt trail is the only way back to where it started.
Each term as this page uses it. Where a standard-setting body defines the term, the wording follows that body.
| Term | Definition |
|---|---|
| Prompt injection | An attack in which instructions hidden in content an AI system reads cause it to act against its operator's intent. |
| Kill switch | A control that halts an agent's actions immediately when triggered. |
| Action receipt | A record of each action an agent took, with its inputs, tool calls and authority. |
Each indicator carries its unit, its cadence, the kind of source behind it and the evidence label a value would carry. No values are shown: a programme supplies them under its own authority.
| Indicator | Unit | Cadence | Source type | Label |
|---|---|---|---|---|
| Active agents with a current authority record | proportion of active agents | weekly | Authority register | DECLARED |
| Tool calls denied at the policy gate | count per agent per period | daily | Policy gate logs | OBSERVED |
| Time from kill-switch activation to halt | seconds | per drill or event | Gate and runtime logs | OBSERVED |
| Agent outputs accepted after human review | proportion of outputs | monthly | Review records | OBSERVED |
| Red-team findings open at promotion | count by severity | per model version | Evaluation records | REPORTED |
| Unsupported claims found in sampled outputs | count per sample | monthly | Reviewer audit | OBSERVED |
| Agent actions with a complete receipt | proportion of actions | daily | Receipt store | OBSERVED |
The topic as a computational graph: named sources feed the methods, an evidence review gates the outputs, and the decisions stay with the authorities that hold them.
Fig. 02Nodes carry their type, two metric rows and a state; connectors show what feeds what; dashed frames name the owners; the timeline is the cadence. Design of the pipeline, not a record of a run.
Named public datasets and institutional records, each with its licence class. Sources by agreement are used only under their terms and never republished.
| Source | What it gives | Licence class |
|---|---|---|
| AI Incident Database | Documented real-world AI failures and harms for hazard identification | Open, terms of the database |
| OECD AI Incidents Monitor | Reported AI incidents and hazards drawn from news sources | Terms of the OECD |
| MITRE ATLAS | Adversary tactics and techniques against AI systems | Free, terms of use |
| OWASP Top 10 for large language model applications | Common vulnerability classes in language model and agent systems | CC BY-SA |
| NIST AI RMF Playbook | Suggested actions for the govern, map, measure and manage functions | Public domain |
| Model cards published with open models | Intended use, evaluation and known limits of a base model | Licence per model |
Nexus Foundry prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.
| Who decides | The question | Evidence this page serves |
|---|---|---|
| Supervising institutions | Whether an agent may act in their project, with which tools and on which data | Authority record, evaluation record, open red-team findings |
| Data owners | Whether their data may be read by agents or used to train or tune models | Data classes in scope; the recorded training decision |
| Reviewers | Whether an agent's draft may be accepted into evidence or a report | Receipts, sources cited and checked, evidence labels |
| AI governance leads and data protection authorities | Whether automated processing meets applicable law and the institution's AI policy | Impact assessment, logs, the human oversight record |
In the design of the Nexus Sovereignty Framework (NSF), every agent and tool holds an authority record that states its purpose, the tools it may call, the data classes it may touch, its rate limits and its kill-switch. Requests without an explicit policy decision fail closed. Agent actions pass the same policy gates and receive the same receipts as human actions, and scopes and kill-switches are enforced at the gate rather than left to the agent's own configuration.
In Labs, agents may run analyses and draft artifacts, but each action is bound to an agent identity, model, tool set, budget and supervising institution, and remains an assertion until a person reviews it. In Studio, agent behaviour inside a composition is declared with its purpose, tools and scope.
Integrations and agents are tested against their declared contracts before they receive authority in any tenant.
Promotion of a model requires evaluation records, including red-team results where the use calls for them.
No tenant data is used for training without a recorded decision by the data owner.
Evaluations sample behaviour and cannot exclude every failure; language models can produce fluent, wrong output; and tool-use risks grow with the number of tools an agent can reach. Narrow scopes and human review are the main mitigations.
Agents never take a decision that belongs to an institution or a competent authority, issue a warning, release funds, dispatch resources, sign an acceptance or present a draft as reviewed evidence.
The current service level is R0: information and enquiries. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.