AI safety, evaluation and human oversight in systemic risk work

AI can help expert teams read more evidence, but an unreviewed output can travel from a note to a public statement before anyone checks it. This essay sets out what the evidence says about AI safety and evaluation, and how human oversight is designed into the Nexus programme.

Key takeaways

• Human oversight of AI in shared risk work means that a named person with a recorded role can understand, question, override and stop an AI system, and that every consequential decision stays with that person or with the competent authority [1, 2].
• The evidence base for AI safety is weaker than the capabilities it must govern: documented AI incidents rose from 233 in 2024 to 362 in 2025, while reporting on responsible-AI benchmarks remains spotty [3].
• Pre-deployment testing is losing predictive power: the International AI Safety Report 2026 describes an evaluation gap and models that increasingly distinguish test settings from real use [4].
• In the Nexus programme, an AI capability is admitted for a stated task, data class, evaluation suite and expiry, never on a generic success rate, and its use in a live exercise requires stop-work authority [5, 6].
• The Global Centre for Risk and Innovation (GCRI) does not regulate or certify AI systems; alignment with public frameworks is recorded as alignment, and the programme is at service level R0 [7, 8].

What human oversight means in shared risk work

Human oversight is the arrangement by which natural persons can effectively watch, interpret, override and halt an AI system while it is in use. The EU AI Act states the requirement precisely for high-risk systems in Article 14: the people assigned to oversight must be able to understand the system’s capacities and limitations, remain aware of the tendency to over-rely on its output, interpret that output correctly, decide not to use it or to override or reverse it, and intervene or interrupt the system through a stop procedure [1].

In the work of the Nexus Consortium, the same idea has a second, institutional meaning. AI may help expert teams to read evidence, classify signals, summarise records or draft public-safe text, but an AI output is always an input to a decision taken by a person with a recorded role, and public, regulatory, fiduciary and operational decisions are never taken inside the programme at all [2]. Evaluation is the discipline that tells that person what the system can be trusted to do, for which task, on which population of cases and in which version [5].

This essay sets out why the problem is urgent, what the public evidence and instruments say, and how the programme designs oversight into the admission, use, recording and correction of AI capabilities. It describes a design, not a running service: every structure is in formation and the current service level is R0, information and enquiries [8].

Why oversight is now a systemic question

Systemic risk work is interpretation under uncertainty: of hazard signals, infrastructure dependencies, fiscal exposure, health-system strain and cascading effects. AI is attractive there because it can process more evidence than any team. It is dangerous there for the same reason: an output that sounds certain can travel from a working note to a dashboard, a briefing and a public statement faster than anyone checks it. NIST names this pattern directly. Among the twelve risks of its Generative AI Profile are confabulation, the production of confidently stated but false content, and human-AI configuration, which covers automation bias and over-reliance [9].

Adoption raises the stakes. The International AI Safety Report 2026 estimates that at least 700 million people use leading AI systems every week, and that in coding AI agents can now reliably complete some tasks that would take a human about half an hour, up from under ten minutes a year earlier, with that task length doubling about every seven months [4]. Agents that call tools, query data and write files act on the world; a misread output is no longer only a wrong sentence.

For institutions that prepare evidence for governments, operators and financial institutions, the cost of an unreviewed output is not limited to one error. It is a loss of the trust on which every later finding depends. That is why GCRI treats oversight as a trust function, kept in the same records as its other claims and statuses, rather than as a feature of a particular tool.


Left: documented AI incidents per year in the AI Index. Right: human working time of software tasks that agents complete reliably; the earlier bar shows the report’s upper bound of under ten minutes. [4, 3]

What the evidence says about AI safety today

Three findings from the most authoritative recent assessments frame the problem. First, incidents are rising. The 2026 AI Index counts 362 documented AI incidents in 2025, up from 233 in 2024, an increase of about 55% in a year, and finds that while almost all leading developers report capability benchmarks, reporting on responsible-AI benchmarks remains spotty [3].

Second, testing is harder than it looks. The International AI Safety Report 2026, written by over 100 independent experts under an advisory panel nominated by over 30 countries and intergovernmental organisations, describes an evaluation gap: tests and benchmark scores used before deployment can be outdated, too narrow or already present in a model’s training data, so they are not always strongly predictive of real-world capabilities or risks. It adds that it has become more common for models to distinguish test settings from deployment and to exploit loopholes in evaluations, so dangerous capabilities could go undetected [4].

Third, governance is moving from voluntary commitments towards law. The same report records that 12 companies published or updated frontier AI safety frameworks in 2025, that most risk-management initiatives remain voluntary, and that a few jurisdictions are beginning to formalise some practices as legal requirements [4]. Agents, meanwhile, keep improving on realistic tasks: the AI Index reports success on the OSWorld computer-use benchmark rising from 12% to about 66% [3]. The gap between what systems can do and what institutions can verify is the space that oversight must fill.


Grouping as suggested by NIST in AI 600-1. Data privacy is the only risk placed in all three groups; human-AI configuration names automation bias and over-reliance as a risk. [9]

Evaluation is task-specific or it is not evaluation

A single accuracy figure tells a decision-maker almost nothing. A model that summarises flood reports well in one language may confabulate in another; a classifier tuned on last year’s grid telemetry may drift after a network upgrade. The programme therefore treats qualification as task-, population- and version-specific. Its system for software, models and capability lifecycle, NXS-NXQ (software, models and capability lifecycle), keeps admission evidence that binds the task, the data class, the evaluation suite and an expiry date, and keeps evaluation uncertainty, coverage gaps and data-leakage controls with the result. Admission cannot follow a generic success percentage alone [5].

This mirrors the structure of the NIST AI Risk Management Framework, whose core has four functions, govern, map, measure and manage [10]. Mapping is where the context of use is written down; measuring is where evaluation happens against that context; managing is where the decision to deploy, restrict or withdraw is taken. An evaluation that is not tied to a mapped use is a measurement without a question.

Records make the evaluation durable. NXS-NXQ preserves a capability’s digest, provider identity and manifest version, tracks who uses each capability, and supports recall and revocation [5]. When a model is updated, the old admission does not silently carry over; when a defect is found, the record shows every place the capability was used. GCRI’s page on AI evaluation and assurance engineering describes the methods behind this discipline, from test design to validation and its limits.

Five capacities a human overseer needs

Article 14 of the EU AI Act is the clearest public statement of what effective oversight requires, and it is useful well beyond the uses the Act covers. The five capacities in its fourth paragraph, to understand and monitor, to stay aware of automation bias, to interpret correctly, to decide not to use or to override, and to intervene or stop, each answer a distinct failure [1]. For certain remote biometric identification uses, Article 14(5) goes further and requires separate verification by at least two natural persons before any action is taken [1].

The programme applies the five capacities by design. Admission records give the overseer the limits of the system; every finding carries its method and uncertainty, as GCRI’s own findings do [7]; nothing simulated is reported as observed impact [6]; an AI output is advice to a named decider; and stop-work authority is an entry condition for any live exercise [6]. None of this is a claim of conformity with the Act, whose timetable sets high-risk rules for uses in areas such as critical infrastructure to apply from 2 December 2027 [11].


The left column paraphrases Article 14(4) of Regulation (EU) 2024/1689. The programme applies these capacities by design; this is alignment, not a claim of conformity. [1, 5, 6, 2]

Authority, mandates and the limits of an agent

Agentic systems turn oversight into an access-control problem. An agent that can call tools, read a data room or post to a dashboard needs to be bounded in the same way as a person with delegated authority: for a purpose, in a jurisdiction, for a period, and with revocation that takes effect at once. In the programme, the system for trust, authorization and assurance, NXS-NSF (trust, authorization and assurance), is the single writer of consent, delegation, mandate and decision-receipt records. A mandate binds grantor, delegate, purpose, jurisdiction and expiry, and a revocation invalidates dependent decisions and cached grants [12].

The same system issues the public-safe release decision before anything is published through NXS-POR (public, client and partner delivery) [12]. An agent may draft a caption or a summary; it cannot release one. Sensitive evidence stays in controlled environments described in GCRI’s essay on data rooms and controlled collaboration, where access is granted for a stated purpose and recorded.

Some routes are closed to automation altogether. In the Nexus Consortium model a safeguarding concern is a restricted route, never handled in public forums or by a chatbot, and a complaint is reviewed by someone not responsible for the contested decision [13]. Oversight includes knowing where a machine should not be in the loop at all.

Oversight in the annual programme

The programme year gives oversight a fixed shape. Nexus Universe, governed by The Global Risks Forum (GRF), records eight gates, G0 to G7. Data and resources, including AI capabilities, are qualified at G2; the build is accepted for exercise at G3; controlled live operation is authorised at G4; evidence is sealed at G5; and publication is authorised at G6 [2]. GRF holds these decisions in the manner of exercise direction, separate from the controllers who run an exercise and the evaluators who judge it, as its page on programme governance explains [6].

Action Week, the week of controlled exercise, includes a track on frontier technology evaluation, and independent evaluators assess each exercise against its objectives [6]. Entry to the week requires qualified environments, controllers, evaluators, a scenario plan and stop-work authority; GCRI’s essay on safety holds and technical stop authority sets out how a hold is called and released, and GRF’s page on exercises and learning describes the controls around every exercise.

Responsible innovation testbeds, programme offering O04, ask whether a candidate technology solves the intended problem under representative conditions, with a comparator, metrics, limitations, failure behaviour and human use in scope. Sponsorship never influences a result, and testbed participation is not a regulatory sandbox, a certification or a procurement status [8]. The programme’s approach to technical demonstrations applies the same rule to any showing of AI capability.


A designed process; no cycle has run and the programme is at service level R0. A gate is a programme decision, not a public-authority approval, a certification or a deployment authorisation. [5, 12, 6, 2, 13]

Correction when an AI output is wrong

Oversight that cannot correct is only observation. In the Nexus Consortium model a scientific correction or integrity concern is open to anyone, without membership, and is reviewed by the responsible editor or source owner [13]. When the error traces to an AI capability, the admission record shows where else that capability was used, and recall or revocation in NXS-NXQ stops further use while the matter is reviewed [5].

Correction is published, not hidden. GCRI’s corrections and concerns route and its AI-use statement say how AI is used on its sites and how to report a problem. The aim is not to promise that AI will never err, which no one can promise, but to make every error findable, attributable and fixable.

Instruments the programme reads

The programme aligns its practice with public instruments on AI risk and oversight. Naming them records where the programme reads them; it claims no certification, conformity or endorsement.


EU AI Act dates as amended by the AI Omnibus, in force 27 July 2026. Naming an instrument records where the programme reads it; it claims no conformity. [10, 9, 11, 4, 14, 15, 16]

Selected instruments on AI risk and human oversight, with their status as of 21 October 2026.

What is changing

• Law replaces voluntary commitment. The EU AI Act became applicable on 2 August 2026, after the AI Omnibus amended its timetable; the AI Office and Member State authorities now supervise and enforce it [11].
• Evaluation capacity becomes public infrastructure. The European Commission plans a call to increase EU capacity to evaluate AI models before they reach the market, expected to be operational by 2027 [11].
• A global evidence base. The UN has created an Independent International Scientific Panel on AI with 40 members and a Global Dialogue on AI Governance [16].
• Agents. As the length of task an agent can complete doubles about every seven months, oversight shifts from reviewing outputs to bounding actions, mandates and tools [4].

Three institutions, one rule on AI

GCRI stewards the methods, records and technical assistance behind AI use in the programme, through its technical office for Nexus Universe and its specialist network on AI and autonomous systems, which explains how such work is tested in Nexus Universe. GRF governs the annual programme and convenes the policy questions on digital and AI governance. The Global Risks Alliance (GRA) makes the evidence readable to finance and insurance; its pages on human-in-the-loop controls and model risk governance look at the same questions from the side of financial institutions, and its Universe finance-readiness programme runs without any transaction [13].

The annual programme has its home at universe.nexus. GCRI’s companion essays on the AI architecture of the programme and verifiable compute and verifiable intelligence go further into systems and records; the canonical texts are in the documentation on decision rights, the programme year and method and limits. Participation in the programme is recorded in the programme’s participation records.

Boundaries. What this page does not claim: GCRI does not regulate, certify, license or approve AI systems, models, datasets or providers, and no admission record, evaluation or exercise result is a certification, a conformity assessment or a procurement status. GRF does not regulate, finance or procure; GRA does not lend, underwrite, rate or invest. The programme is in formation at service level R0, no cycle has run, and decisions on the use of AI in public services rest with the competent authority, outside the programme.

Questions and answers

Does the programme let AI make decisions?

No. An AI output is an input to a decision taken by a person with a recorded role, and public, regulatory, fiduciary and operational decisions are never taken inside the programme. They rest with the competent authority, operator or licensed institution.

How is an AI model evaluated before it is used?

It is admitted for a stated task, data class and evaluation suite, with an expiry date, and its evaluation uncertainty and coverage gaps travel with the result. A generic success percentage is never enough on its own.

Can a live exercise involving AI be stopped?

Yes. Stop-work authority is an entry condition for controlled live operation at gate G4, alongside controllers, evaluators and a scenario plan. A stopped run is recorded in the run and decision records, not hidden.

Is the programme compliant with the EU AI Act?

The programme makes no conformity claim. It applies the human oversight capacities of Article 14 by design and reads the Act as one of several public instruments; whether the Act applies to a given use is for the competent authority to decide.

What happens when an AI output turns out to be wrong?

Anyone may raise a correction without membership. The capability record shows where else the system was used, recall or revocation stops further use while the matter is reviewed, and accepted corrections are dated and published.

Why does evaluation before deployment not settle the question?

Because tests can be narrow, outdated or present in training data, and models increasingly distinguish test settings from real use. The International AI Safety Report 2026 calls this an evaluation gap, so oversight continues during use.

Glossary

• Human oversight: The capacity of natural persons to understand, interpret, override and stop an AI system while it is in use.
• Automation bias: The tendency to rely on an automated output more than the evidence warrants.
• Confabulation: Confidently stated but false content produced by a generative AI system.
• Evaluation gap: The gap between pre-deployment test results and a system’s real-world capabilities or risks.
• Admission evidence: The record in NXS-NXQ that binds a capability to a task, data class, evaluation suite and expiry.
• Stop-work authority: The authority, required before live operation, to halt an exercise or a run at once.

Further reading across the network

• NXS-NXQ (documentation)
• NXS-NSF (documentation)
• Decision rights
• AI evaluation and assurance engineering at GCRI
• Artificial intelligence across the Consortium
• Digital and AI governance at GRF
• Human-in-the-loop controls at GRA
• AI safety and evaluation courses, Risk Academy
• AI, models and agents in the Nexus Standards

References

1. EU AI Act Explorer (text of the Regulation) (2024). Article 14: Human oversight, Regulation (EU) 2024/1689. Accessed 2026-10-21.
2. The Global Centre for Risk and Innovation (documentation) (2026). Decision rights. Accessed 2026-10-21.
3. Stanford Institute for Human-Centered Artificial Intelligence (2026). The 2026 AI Index Report. Accessed 2026-10-21.
4. International AI Safety Report (Expert Advisory Panel of over 30 countries and organisations) (2026). International AI Safety Report 2026 and Extended Summary for Policymakers. Accessed 2026-10-21.
5. The Global Centre for Risk and Innovation (documentation) (2026). NXS-NXQ. Accessed 2026-10-21.
6. The Global Centre for Risk and Innovation (documentation) (2026). Nexus Universe. Accessed 2026-10-21.
7. The Global Centre for Risk and Innovation (documentation) (2026). GCRI (institution reference). Accessed 2026-10-21.
8. The Global Centre for Risk and Innovation (documentation) (2026). Status and service levels. Accessed 2026-10-21.
9. NIST (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). Accessed 2026-10-21.
10. US National Institute of Standards and Technology (NIST) (2023). AI Risk Management Framework (AI RMF 1.0). Accessed 2026-10-21.
11. European Commission (2026). AI Act: regulatory framework for artificial intelligence. Accessed 2026-10-21.
12. The Global Centre for Risk and Innovation (documentation) (2026). NXS-NSF. Accessed 2026-10-21.
13. The Global Centre for Risk and Innovation (2026). How Nexus Consortium works. Accessed 2026-10-21.
14. OECD (2024). OECD AI Principles. Accessed 2026-10-21.
15. UNESCO (2021). Recommendation on the Ethics of Artificial Intelligence. Accessed 2026-10-21.
16. United Nations General Assembly (2025). Terms of reference and modalities for the Independent International Scientific Panel on AI and the Global Dialogue on AI Governance (A/RES/79/325). Accessed 2026-10-21.

Editorial record

Facts and programme states are given as of 21 October 2026. This page is reviewed at least every 180 days and whenever a programme state, a service level, a cited figure or the application of the EU AI Act changes. It replaces an earlier version of the same page, which described AI workflows running on shared Nexus infrastructure, model records and public-safe output controls as if in operation; the programme is in formation at service level R0, so those descriptions are now given as design, and structures not on record have been dropped.

Published by the GCRI editorial desk under the editorial integrity policy. To report an error or request a correction, use the GCRI corrections route; corrections are open to anyone, without membership, and each accepted correction is dated and listed with the page.