Artificial Intelligence and Autonomous Systems
The Global Centre for Risk and Innovation (GCRI)
Agentic AI systems pursue goals by planning and taking actions: calling tools, browsing, writing and running code, or operating other software, sometimes over long tasks with little supervision. Autonomy brings usefulness and new failure modes, from compounding errors to misuse of granted permissions. This page sets out how agents fail, how their actions can be bounded and overseen, and what evidence serves those who build, deploy and govern them.
GCRI · In development · service level R0
Agentic AI is AI that pursues goals by planning and taking actions with tools and software, often over many steps with little supervision. Its risks come from compounding errors, permissions that set the scope of harm and untrusted inputs that steer behaviour, so oversight and limits must be designed in.
Night-time lights and population density, the footprint of electrified, connected activity, framed on East Asia; each layer is a dated NASA product.
The interactive map needs JavaScript; the layer list and sources are below.
Layers 2
Connected activity
Keyboard, with the map focused
Hold Ctrl or ⌘ and scroll to zoom
Latest available
NASA GIBS / EOSDIS · ASTER GDEM (METI, NASA) · Natural Earth
Loading the latest NASA imagery.
Drag to move. Ctrl or ⌘ and scroll, pinch or the buttons to zoom; hover for the values under the pointer; reset returns to the region.
Night-time lights and population density, the footprint of electrified, connected activity, framed on East Asia.Satellite and model products as NASA publishes them, each with its own date; indicators, not assessments of any place; claim ceiling INTEGRATED.
Layers and dates
The working screen behind this subject: its structure, terms and states. Names are placeholders and values illustrative.
Agentic systems · Agent sandbox · Session AG-6627
Tool access, delegated authority and safe execution. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.
Each step in a long task can go wrong, and an early mistake shapes later actions. Small per-step error rates add up over many steps.
An agent can only do harm with the access it has been given. Broad credentials, write access and payment rights turn errors or manipulation into real-world consequences.
Agents read web pages, documents and messages that may contain instructions. Prompt injection can redirect an agent to act against its user.
Humans cannot review every action of a fast agent. Approval steps for consequential actions, logging and the ability to stop and roll back are design choices.
Each term as this page uses it. Where a standard-setting body defines the term, the wording follows that body.
| Term | Definition |
|---|---|
| Excessive agency | Granting an AI system more permissions or autonomy than its task requires, an OWASP risk category. |
| Tool use | An AI system's ability to call external functions, services or software to act. |
| Human in the loop | A design in which a person approves or can intervene in an AI system's actions. |
Each indicator carries its unit, its cadence, the kind of source behind it and the evidence label a value would carry. No values are shown: a programme supplies them under its own authority.
| Indicator | Unit | Cadence | Source type | Label |
|---|---|---|---|---|
| Task success rate on agent benchmarks | percent of tasks completed correctly | per evaluation | Agent benchmark results | REPORTED |
| Actions requiring human approval | percent of consequential actions | per deployment | Deployment configuration | DECLARED |
| Prompt injection success rate | percent of attempts succeeding | per evaluation | Security testing | OBSERVED |
| Agent incidents | count by severity | monthly | Incident reports | REPORTED |
| Actions logged and attributable | percent of agent actions | per deployment | Audit logs | OBSERVED |
| Mean task length before human intervention | steps or minutes | per evaluation | Evaluation logs | OBSERVED |
The topic as a computational graph: named sources feed the methods, an evidence review gates the outputs, and the decisions stay with the authorities that hold them.
Fig. 02Nodes carry their type, two metric rows and a state; connectors show what feeds what; dashed frames name the owners; the timeline is the cadence. Design of the pipeline, not a record of a run.
Named public datasets and institutional records, each with its licence class. Sources by agreement are used only under their terms and never republished.
| Source | What it gives | Licence class |
|---|---|---|
| OWASP Top 10 for LLM Applications and agentic security guidance | Risks such as prompt injection and excessive agency | Open, CC BY-SA 4.0 |
| Published agent benchmarks such as SWE-bench and related suites | Task-based evaluation of agents | Open, licence per benchmark (to check) |
| NIST AI Risk Management Framework | Practices for governing and measuring AI risk | Public domain, US government |
| AI Incident Database | Reported incidents involving AI systems | Open, licence to check |
| Developer system cards describing agent safeguards | Developer-reported agent evaluations and controls | Public documents, terms per developer |
Artificial Intelligence and Autonomous Systems prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.
| Who decides | The question | Evidence this page serves |
|---|---|---|
| Deploying organizations | Which actions an agent may take alone and which need approval | Permission patterns; evaluation results |
| Security teams | How to protect agents from manipulation | Injection testing; incident evidence |
| Regulators | What oversight to require for autonomous AI actions | Oversight design evidence |
An agent combines a model's reasoning with permissions in the world. Small errors in early steps can compound across a long task. Agents can misinterpret goals, pursue them in unintended ways or take shortcuts that satisfy a metric but not the intent. Content the agent reads, such as a web page or document, can contain instructions designed to hijack it, an attack known as prompt injection.
The more permissions an agent holds, the larger the potential harm. Systems that coordinate several agents add further failure modes, as agents pass errors to each other or interact in unforeseen ways.
Granting only the permissions a task needs, with sandboxes and spending or action limits, drawing on cybersecurity and cryptography.
Requiring confirmation for consequential or irreversible actions. Too many prompts lead to rubber-stamping, so placement matters.
Testing agents on realistic multi-step tasks in controlled environments, using AI evaluation and assurance engineering. Simulated environments miss some real-world complexity.
Recording actions and reasoning traces so incidents can be reconstructed. Traces may not fully reflect how a model reached a decision.
The current service level is R0: information and enquiries. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.