Skip to content

GCRI · Zîrekiya çêkirî û pergalên xweser

Statistics for AI evaluation

Probability and statistics make AI evaluation trustworthy: confidence intervals on scores, significance of differences between models, calibration of predicted probabilities and sampling designs for audits. In this field they stop small test sets and noisy metrics from being over-read.

Pirsek bike Back to STEM foundation of Artificial Intelligence and Autonomous Systems
01Qeyd

Statistics for AI evaluation on the record.

Method recordSTEM method
  1. RêbazÎhtîmal û statîstîk
  3. Applied inZîrekiya çêkirî û Pergal
  6. Canonical pageSTEM Commons
02Application

What probability and statistics does here.

Probability and statistics provide the mathematical framework for describing uncertainty and learning from data. They underpin estimation, testing and prediction in every risk discipline. In artificial intelligence and data and compute, statistical methods quantify the likelihood and uncertainty of losses across interconnected pergalan. They make the evidence behind risk decisions explicit and reproducible. The canonical description is on STEM Commons.

Uncertainty on benchmark scores

Bootstrap and analytical intervals are reported with every score, and differences between models are tested before rankings are stated.

Calibration assessment

Predicted probabilities are compared with observed outcomes to check whether a model’s confidence matches its accuracy.

Audit sampling

Samples of decisions are drawn with stated designs so that error rates for subgroups can be estimated with known precision.

03Rêbaz û delîl

How Statistics for AI evaluation is modelled, measured and tested.

Model, texmînên wî, daneyên wî û nediyariya wî, bi şeweyê ku zanyar an endezyar kontrol bike. Avahî rêbazên weşandî dişopînin. Nirxên hatine xêzkirin nimûneyî ne, ne encam in.

Model, hevkêşe û texmîn

The model structure used for Statistics for AI evaluation, the assumptions it makes and how each is tested.

Xetera hêmana iRi = f(Hi, Ei, Vi, Ci)metirsî, rûbirûbûn, lawazî û kapasîteya serederiyê
Windahiya salane ya hêvîkirîEAL = ∑k pk · Lkli ser bûyerên k bi îhtîmala salane pk
Pergalên girêdayîxt+1 = A xt + B ut + εtA carries the couplings, u the shock, ε the error
TexmînÇima tê çêkirinÇawa tê ceribandin
Metirsî û rûbirûbûn di nav bûyerekê de serbixwe neEntegrala windahiyê ji hesabkirinê re maqûl dihêleBi lihevhatineke copula li ser tomarên hevbeş re bidin ber hev
Girêdan li nêzî xala xebatê xêzik inDihêle şok di A de belav bibinJi derveyî rêza xêzik tevlihev bike û bide ber hev
Xêzên lawaziyê di navbera cihan de têne veguhastinXêzikên herêmî kêm tên dîtinLi hember windahiyên çavdêrîkirî erê bike li cihê ku hene
Têketin
danehevên bi guhertoyan û çavkaniyê
Derketin
belavkirin, tu carî jimarên tenê
Kod
bi qerta modelê û ceribandinan re hat weşandin
Zêdebûn
heman dane, kod, jîngeh û tov
Formulasyonên standard ên xetereyê û qada rewşê; texmînên navborî ji bo rêbazê tîpîk in, ji bo nirxandinê hatine diyarkirin, ne encam in.
Prensîbên rêber ên FAIR · GO FAIR û FORCE11W3C PROV · World Wide Web ConsortiumRêwerzên ji bo birêvebirina hevgirtî ya nediyarîyan · IPCCCEOS Analysis Ready Data · Committee on Earth Observation Satellites
04Dane

What the method consumes.

Datasets and records used by probability and statistics on this platform, with the role each plays and the label its values carry. Sources are listed in the source register.

DaneRolNîşan
Evaluation outputsScore estimationÇAVDÊRÎKIRÎ
Labelled audit samplesError estimationÇAVDÊRÎKIRÎ
Prediction logsCalibrationÇAVDÊRÎKIRÎ
05Pêvajo

Pêvajo çawa dimeşe.

Process explorer · Process

Statistics for AI evaluation: Catastrophe model pipeline

10 steps5 qonax

Pêvajo

Statistics for AI evaluation: Catastrophe model pipeline

Catastrophe models usually have hazard, vulnerability, exposure and financial modules, and produce loss tables, windahiya salane ya navînî, exceedance curves, PML and TVaR.

  1. 01 Hazard

    1. 01Event catalogueDataset · Model developerThe hazard module holds a large catalogue of simulated events that represents a wide range of plausible scenarios.
    2. 03Hazard moduleModel · Model developerFor each event the hazard module gives the local hazard intensity that the vulnerability module reads.

  2. 02 Exposure and damage

    1. 02RûbirûbûnDataset · Incident responderThe exposure module houses the portfolio data, such as location-specific information.
    2. 04Vulnerability moduleModel · Model developerThe vulnerability module quantifies the expected damage to the exposures from an event.

  3. 03 Financial

    1. 05Financial moduleModel · Incident responderThe financial module calculates losses from all event scenarios. Ground-up, gross and net losses differ as shown below.
    2. 06Event loss tableDataset · Incident responderAn event loss table lists each event with its annual occurrence probability and its loss for the portfolio.
    3. 07Year loss tableDataset · Incident responderSome models also sample simulated years from event losses. This outline does not describe their format.

  4. 04 Metrics

    1. 08Exceedance probability curvesModel · Incident responderThe occurrence curve gives the chance that the largest loss in a year exceeds an amount. The aggregate curve does so for the sum of losses.
    2. 09AAL, PML and TVaRModel · Incident responderAverage annual loss sums event probability times loss. PML is one point on the occurrence curve.

  5. 05 Use of results

    1. 10Use of resultsDecision · Incident responderThe tail metrics are used when considering a portfolio’s exposure and the level of catastrophe reinsurance an insurer requires.

Outline of an external standard or method, from its public sources. It describes what the framework says, not a record of any run. It does not show that Nexus or any partner performs, certifies or approves a step. A gate is a step of the method, never a Nexus approval.

06Erêkirin

How results are checked.

Results are checked through diagnostics, cross-validation and sensitivity to assumptions. Independent replication confirms findings.

Kontrol bikinHow it is done
Interval reportingEvery metric is reported with its uncertainty
Multiple comparisonsComparisons across many tasks are corrected for multiple testing
Sample size planningAudit samples are sized to detect differences of stated magnitude

Inference depends on assumptions about independence, distribution and sampling that may not hold. Small or biased samples limit reliability.

07Connections

Topics that rely on this method.

08Stewardship

Who maintains the method.

Zanista pergalan û modelkirin stewards probability and statistics for the Consortium and reviews how it is applied across platforms.

Canonical page: Probability and statistics

09Girêdayî

Ev rûpel li ser Konsorsiyûmê te ber bi kû ve dibe.

10Pirs

Pirs û bersiv.

What are the limits of probability and statistics in this field?
Inference depends on assumptions about independence, distribution and sampling that may not hold. Small or biased samples limit reliability.
How are results checked?
Results are checked through diagnostics, cross-validation and sensitivity to assumptions. Independent replication confirms findings.
Are the values on this page real?
The data names, roles and checks are real descriptions of practice. The canvas shows a design of the method’s use, not a run; no result is reported here.
Where else is this method applied?
Çavdêrxaneya canonical method page lists every platform that applies it.
rêzeya Pergalên StratejîkTevkarî bike

Ji vê rûpelê delîl bidin.

rêzeya Pergalên Stratejîk is the Consortium’s quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.

GCRIZîrekiya çêkirî û pergalên xweserTeknolojiya eksponensiyelDelîlên Labs

Tu carî daneyên şexsî, hûrgiliyên binesaza hesas an tiştekî ku mafê we yê parvekirinê tune ne neşînin. Beşdarî berî bikaranînê tê nirxandin; şandina wê nivîskarî an mafên edîtorî nade û sponsorî tu carî encamekê nakire.

Zîrekiya çêkirî û pergalên xweser

Rêbazek, danehevek an pirsek bîne.

Agahî û pirs îro vekirî ne. Pirsek referansekê distîne û gava piştî wê dide nasîn; serlêdan, drav, veqetandina hesabkirinê û tomarkirin tenê dema ku tîm, mercên wê û kapasîteya vekolînê hebin vedibin.

Rêbazekê pêşniyar bike Hemû rêyên têkiliyê bibînin
Hiqûqî

Nexşe, dane û jimarên vê malperê ji bo çarçoveyê û agahiyên giştî têne nîşandan. Ew ne şîret, ne pêşbînî û ne hişyarî ne; sînorên agahiyê di Daxuyaniyên Berpirsiyariyê û Daxuyaniyên Daneyê de hatine diyarkirin.