GCRI · 人工智能与自主系统
智能体系统
Agentic AI 系统 pursue goals by planning and taking actions: calling tools, browsing, writing and running code, or operating other software, sometimes over long tasks with little supervision. Autonomy brings usefulness and new failure modes, from compounding errors to misuse of granted permissions. This page sets out how agents fail, how their actions can be bounded and overseen, and what evidence serves those who build, deploy and govern them.
Agentic systems: how the risk develops.
Tool access, delegated authority and safe execution. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.
Errors compound over steps
Each step in a long task can go wrong, and an early mistake shapes later actions. Small per-step error rates add up over many steps.
Permissions define the blast radius
An agent can only do harm with the access it has been given. Broad credentials, write access and payment rights turn errors or manipulation into real-world consequences.
Untrusted inputs steer behaviour
Agents read web pages, documents and messages that may contain instructions. Prompt injection can redirect an agent to act against its user.
Oversight must be designed in
Humans cannot review every action of a fast agent. Approval steps for consequential actions, logging and the ability to stop and roll back are design choices.
Agent oversight console.
团队跟踪该主题时会使用的界面：其指标、状态与决策。名称为示例，数值仅作示意。
Agent oversight console
Agentic systems · Agent sandbox · Session AG-6627
Tool calls
- 10:34信息search(documents) · read-only
- 10:51信息run_model(flood_nowcast) · sandbox
- 10:51关口send_message(external) · needs approval
- 10:53Errorwrite_file(outside workspace) · denied
政策
- 范围
- workspace only
- External actions
- human approval
- Spend limit
- 43 units
- Kill switch
- Armed
Actions by approval class
Agents that act need guardrails in every domain they touch.
Agentic system studies are designed for the 指数级技术系列. The OWASP Top 10 for LLM applications lists failure modes such as excessive agency and prompt injection; the page describes them without exploit detail.
- Incident reports involving agents.
- Designs for oversight and permission scoping.
How agentic systems is modelled, measured and tested.
模型、其假设、数据和不确定性，以科学家或工程师会核查的形式呈现。结构遵循已发表的方法。所取数值仅为示意，不是研究发现。
系统模型
The couplings between the systems in agentic systems: signs of influence, a reinforcing and a balancing loop, and cross-links where stress jumps.
衡量什么，以及衡量得如何。
每个指标都注明其单位、频率、背后的来源类型以及数值将附带的证据标签。此处不显示数值：数值由计划依据其自身权限提供。
| 指标 | 单位 | 节奏 | 来源类型 | 标签 |
|---|---|---|---|---|
| Task success rate on agent benchmarks | percent of tasks completed correctly | per evaluation | Agent benchmark results | 已报告 |
| Actions requiring human approval | percent of consequential actions | per deployment | Deployment configuration | 已声明 |
| Prompt injection success rate | percent of attempts succeeding | per evaluation | Security testing | 观测 |
| Agent incidents | count by severity | 每月 | Incident reports | 已报告 |
| Actions logged and attributable | percent of agent actions | per deployment | Audit logs | 观测 |
| Mean task length before human intervention | steps or minutes | per evaluation | Evaluation logs | 观测 |
证据来自何处。
具名公共数据集与机构记录，各自注明许可类别。依协议获得的来源仅按其条款使用，绝不再次发布。
| 来源 | 它提供什么 | 许可类别 |
|---|---|---|
| OWASP Top 10 for LLM Applications and agentic security guidance | Risks such as prompt injection and excessive agency | Open, CC BY-SA 4.0 |
| Published agent benchmarks such as SWE-bench and related suites | Task-based evaluation of agents | Open, licence per benchmark (to check) |
| NIST 人工智能风险管理框架 | Practices for governing and measuring AI risk | Public domain, US government |
| AI Incident Database | Reported incidents involving AI 系统 | Open, licence to check |
| Developer system cards describing agent safeguards | Developer-reported agent evaluations and controls | Public documents, terms per developer |
由谁决定，以及证据服务于什么。
人工智能与自主 系统 prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.
| 谁来决定 | 问题 | 本页面服务的证据 |
|---|---|---|
| Deploying organisations | Which actions an agent may take alone and which need approval | Permission patterns; evaluation results |
| Security teams | How to protect agents from manipulation | Injection testing; incident evidence |
| 监管机构 | What oversight to require for autonomous AI actions | Oversight design evidence |
该主题产生的产出
- Permission and approval design patterns for agents
- Agent evaluation protocols covering long tasks
- Prompt injection test suites for deployments
- Logging and rollback requirements
What designers and evaluators ask.
- What actions can the agent take, with whose authority, and which are reversible?
- How does it behave when instructions are ambiguous, conflicting or malicious?
- Can a human see what it is doing, understand why and stop it in time?
- How do errors compound over long sequences of steps?
How agents fail.
An agent combines a model’s reasoning with permissions in the world. Small errors in early steps can compound across a long task. Agents can misinterpret goals, pursue them in unintended ways or take shortcuts that satisfy a metric but not the intent. Content the agent reads, such as a web page or document, can contain instructions designed to hijack it, an attack known as prompt injection.
The more permissions an agent holds, the larger the potential harm. 系统 that coordinate several agents add further failure modes, as agents pass errors to each other or interact in unforeseen ways.
Designing for safe execution.
- 01
Least privilege
Granting only the permissions a task needs, with sandboxes and spending or action limits, drawing on 网络安全与密码学.
- 02
Human oversight points
Requiring confirmation for consequential or irreversible actions. Too many prompts lead to rubber-stamping, so placement matters.
- 03
Task-based evaluation
Testing agents on realistic multi-step tasks in controlled environments, using AI 评估与保证工程. Simulated environments miss some real-world complexity.
- 04
Logging and audit
Recording actions and reasoning traces so incidents can be reconstructed. Traces may not fully reflect how a model reached a decision.
Where this page’s references lead.
阅读本页时所对照的机构和文件。状态和版本会变化；每个链接都指向发布机构。
问题与解答。
What is an AI agent?
What is prompt injection?
How can agent risk be reduced?
Should agents ever act without human approval?
Can agents be evaluated before deployment?
Does the platform build agents?
从本页贡献证据。
报告 is the Consortium’s quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.
- 电子邮件网关 GW-GCRI从本页发送证据、更正、案例、方法、审查、翻译或概况投稿。它会带上本页标签送达，由负责的编辑接收。
- Zenodo开放存缴存缴并提交到社区用您的 ORCID iD 在 Zenodo 上发布您自己的记录，然后提交到那里的 Reports 社区。由策展人审核；DOI 仍归您所有。
- 报告GCRI查看所有贡献方式贡献类型、各自的评审方式、您获得的署名、许可协议，以及绝不接受的内容。
切勿发送个人数据、敏感基础设施细节或您无权分享的任何内容。贡献在使用前会经过审查；发送贡献不赋予署名权或编辑权，赞助也绝不能换取研究结论。
法律文件。
本网站上的地图、数据和图表仅作为背景和一般信息展示。它们不是建议、预测或预警；信息的局限见免责声明和数据说明。