Skip to content

GCRI · 人工智能与自主系统

面向人工智能评估的统计

Probability and statistics make AI evaluation trustworthy: confidence intervals on scores, significance of differences between models, calibration of predicted probabilities and sampling designs for audits. In this field they stop small test sets and noisy metrics from being over-read.

提出咨询 Back to STEM foundation of Artificial Intelligence and Autonomous Systems
01登记

Statistics for AI evaluation on the record.

方法记录STEM 方法
  1. 方法概率与统计
  3. 应用于人工智能与自主 系统
  4. 数据3 inputs
  6. 规范页面STEM 公共资源
02申请

What probability and statistics does here.

Probability and statistics provide the mathematical framework for describing uncertainty and learning from data. They underpin estimation, testing and prediction in every risk discipline. In artificial intelligence and data and compute, statistical methods quantify the likelihood and uncertainty of losses across interconnected 系统. They make the evidence behind risk decisions explicit and reproducible. The canonical description is on STEM Commons.

Uncertainty on benchmark scores

Bootstrap and analytical intervals are reported with every score, and differences between models are tested before rankings are stated.

Calibration assessment

Predicted probabilities are compared with observed outcomes to check whether a model’s confidence matches its accuracy.

Audit sampling

Samples of decisions are drawn with stated designs so that error rates for subgroups can be estimated with known precision.

03方法与证据

How Statistics for AI evaluation is modelled, measured and tested.

模型、其假设、数据和不确定性，以科学家或工程师会核查的形式呈现。结构遵循已发表的方法。所取数值仅为示意，不是研究发现。

模型、方程与假设

The model structure used for Statistics for AI evaluation, the assumptions it makes and how each is tested.

要素 i 的风险Ri = f(Hi, Ei, Vi, Ci)危害、风险暴露、脆弱性和应对能力
预期年度损失EAL = ∑k pk · Lk对年概率为 pk
耦合系统xt+1 = A xt + B ut + εtA carries the couplings, u the shock, ε the error
假设为何制作如何检验
在一次事件中，危害与暴露相互独立使损失积分保持可计算与基于联合记录拟合的 copula 进行比较
耦合在工作点附近是线性的让冲击通过 A 传播在线性范围之外施加扰动并比较
脆弱性曲线可以在不同地点之间迁移使用很少有本地曲线可用在有观测损失的地方据此进行验证
输入
带溯源信息的版本化数据集
成果
分布，绝不是单一数字
代码
连同模型卡和测试一起发布
复现
相同的数据、代码、环境和随机种子
标准的风险与状态空间表述；所列假设是该方法的典型假设，供审查之用，并非结果。
FAIR 指导原则 · GO FAIR 和 FORCE11W3C PROV · 万维网联盟关于一致处理不确定性的指南 · IPCCCEOS 分析就绪数据 · 地球观测卫星委员会
04数据

该方法使用的输入。

Datasets and records used by probability and statistics on this platform, with the role each plays and the label its values carry. Sources are listed in 来源登记册.

数据角色标签
Evaluation outputsScore estimation观测
Labelled audit samplesError estimation观测
Prediction logs校准观测
05流程

流程如何运行。

Process explorer · Process

Statistics for AI evaluation: Catastrophe model pipeline

10 steps5 个阶段

流程

Statistics for AI evaluation: Catastrophe model pipeline

巨灾模型通常包括危害、脆弱性、风险暴露与财务模块，并生成损失表、 年均损失、超越曲线、PML 与 TVaR。

  1. 01 危害

    1. 01事件目录数据集 · 模型开发者危害模块包含一个大型模拟事件目录，代表多种多样的合理情景。
    2. 03危害模块模型 · 模型开发者对于每个事件，危害模块给出当地的 危害强度 ，供脆弱性模块读取。

  2. 02 风险暴露与损害

    1. 02风险暴露Dataset · Incident responder风险暴露模块存放组合数据，例如特定地点的信息。
    2. 04脆弱性模块模型 · 模型开发者脆弱性模块量化某一事件对风险暴露对象造成的预期损害。

  3. 03 财务

    1. 05财务模块Model · Incident responder财务模块计算所有事件情景造成的损失。基础损失、总损失与净损失的区别如下所示。
    2. 06事件损失表Dataset · Incident responder事件损失表列出每个事件的年发生概率及其对组合造成的损失。
    3. 07年度损失表Dataset · Incident responder有些模型还会从事件损失中抽样模拟年份。本概述不描述其格式。

  4. 04 指标

    1. 08超越概率 曲线Model · Incident responder发生曲线给出一年中最大损失超过某一金额的概率。累积曲线针对损失总和给出同样的概率。
    2. 09AAL、PML 与 TVaRModel · Incident responder年均损失是事件概率与损失乘积之和。PML 是发生曲线上的一个点。

  5. 05 结果的使用

    1. 10结果的使用Decision · Incident responderThe tail metrics are used when considering a portfolio’s exposure and the level of catastrophe reinsurance an insurer requires.

依据公开来源对某一外部标准或方法的概述。它描述的是该框架的内容，而非任何运行记录。它并不表明 Nexus 或任何合作伙伴执行、认证或批准某一步骤。关口是该方法的一个步骤，绝不是 Nexus 的批准。

06验证

结果如何核查。

Results are checked through diagnostics, cross-validation and sensitivity to assumptions. Independent replication confirms findings.

核查如何开展
Interval reportingEvery metric is reported with its uncertainty
Multiple comparisonsComparisons across many tasks are corrected for multiple testing
Sample size planningAudit samples are sized to detect differences of stated magnitude

Inference depends on assumptions about independence, distribution and sampling that may not hold. Small or biased samples limit reliability.

07关联

依赖该方法的主题。

08管理责任

由谁维护该方法。

系统科学与建模 stewards probability and statistics for the Consortium and reviews how it is applied across platforms.

Canonical page: Probability and statistics

09已关联

本页在整个联合体中通向何处。

10问题

问题与解答。

What are the limits of probability and statistics in this field?
Inference depends on assumptions about independence, distribution and sampling that may not hold. Small or biased samples limit reliability.
结果如何核查？
Results are checked through diagnostics, cross-validation and sensitivity to assumptions. Independent replication confirms findings.
本页面上的数值是真实的吗？
The data names, roles and checks are real descriptions of practice. The canvas shows a design of the method’s use, not a run; no result is reported here.
该方法还应用于何处？
维持社会运转的 规范方法页面 列出应用该方法的每个平台。
报告贡献

从本页贡献证据。

报告 is the Consortium’s quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.

GCRI人工智能与自主系统指数级技术Labs 证据

切勿发送个人数据、敏感基础设施细节或您无权分享的任何内容。贡献在使用前会经过审查；发送贡献不赋予署名权或编辑权，赞助也绝不能换取研究结论。

人工智能与自主系统

带来一种方法、一个数据集或一个问题。

信息和咨询现已开放。每项咨询都会获得一个编号，并告知下一步；申请、付款、算力分配和注册只有在相应团队、条款和审查能力就绪时才会开放。

提出一种方法 查看全部联系途径
法律

本网站上的地图、数据和图表仅作为背景和一般信息展示。它们不是建议、预测或预警；信息的局限见免责声明和数据说明。