The Global Centre for Risk and Innovation (GCRI)
Machine learning now enters science at every step: screening literature, designing experiments, analysing images and sequences, emulating simulations and drafting text. Each use can speed discovery and each can introduce errors that look like findings. This page sets out how models are evaluated for scientific use, where leakage, benchmark overfitting and silent drift arise, and what evidence editors, funders and regulators need before they rely on a model-assisted result.
AI in scientific workflows · Field lab coast · Campaign 5
placeholder coastline
| S-01 tide | 94 | 0.0 | Nominal |
| S-04 salinity | 76 | 0.4 | Recalibrate |
| S-07 wave | 95 | 0.1 | Nominal |
Fig.Testbed operations. Design wireframe with placeholder names and illustrative values, not a record or a live system.
AI in scientific research is the use of machine learning to screen literature, design experiments, analyse images and sequences, emulate simulations and draft text. Each use can speed discovery and each can introduce errors that look like findings, through data leakage, overfitted benchmarks, drift or fabricated references.
For ai and scientific workflows, in GCRI's spatial-intelligence reading, the map opens on true-colour imagery, night lights and population density with true colour from NASA GIBS, framed on World; every other NASA layer, live feed and forecast waits in the layer list.
The interactive map needs JavaScript; the layer list and sources are below.
Layers 3
Connected activity, for this page
Keyboard, with the map focused
Hold Ctrl or ⌘ and scroll to zoom
Latest available
NASA GIBS / EOSDIS · ASTER GDEM (METI, NASA) · Blue Marble (NASA) · © OpenStreetMap contributors (ODbL) · Natural Earth
Loading the latest NASA imagery.
Drag to move. Ctrl or ⌘ and scroll, pinch or the buttons to zoom; hover for the values under the pointer; reset returns to the region.
For ai and scientific workflows, in GCRI's spatial-intelligence reading, the map opens on true-colour imagery, night lights and population density with true colour from NASA GIBS, framed on World; every other NASA layer, live feed and forecast waits in the layer list.Satellite and model products as NASA publishes them, each with its own date; indicators, not assessments of any place; claim ceiling INTEGRATED.
Layers and dates
Model evaluation and scientific assistance. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.
When information from the test set reaches training, through duplicates, shared patients, neighbouring sites or preprocessing fitted on all the data, a model appears more accurate than it is. Leakage has been documented in many fields that adopted machine learning.
Repeated tuning against the same public benchmark selects models that fit its quirks. Gains on a leaderboard can vanish on new data, and contamination of training corpora with benchmark items makes scores hard to interpret.
Instruments, protocols, populations and software change. A model validated on last year's data or another lab's scanner can degrade without any visible error. Monitoring after deployment is part of the evaluation, not an extra.
Language models can produce fluent text, citations and code that are wrong. Unchecked, these errors enter manuscripts, peer reviews and proposals, and spread through citation. Disclosure and verification policies exist because such errors are hard to see.
Each term as this page uses it. Where a standard-setting body defines the term, the wording follows that body.
| Term | Definition |
|---|---|
| Data leakage | Information from the test set reaching a model during training, which inflates reported performance. |
| Benchmark overfitting | Tuning methods to a fixed benchmark until scores no longer reflect performance on new data. |
| Distribution shift | A change between the data a model was trained on and the data it meets in use. |
Each indicator carries its unit, its cadence, the kind of source behind it and the evidence label a value would carry. No values are shown: a programme supplies them under its own authority.
| Indicator | Unit | Cadence | Source type | Label |
|---|---|---|---|---|
| Held-out evaluation design stated | percent of model-based studies | annual | Audit against the REFORMS checklist | OBSERVED |
| Code and model weights available | percent of studies with runnable code and weights | annual | Repository and article audit | OBSERVED |
| External validation | percent of models tested on data from another site or period | annual | Literature review | ESTIMATED |
| Calibration error | expected calibration error, dimensionless | per model | Held-out evaluation | OBSERVED |
| Performance gap under shift | difference in the primary metric between internal and external test sets | per model | Validation reports | OBSERVED |
| AI use disclosure | percent of submissions with a disclosure statement | annual | Journal submission records | DECLARED |
| Contamination checks reported | percent of benchmark evaluation reports | annual | Evaluation reports | REPORTED |
The topic as a computational graph: named sources feed the methods, an evidence review gates the outputs, and the decisions stay with the authorities that hold them.
Fig. 02Nodes carry their type, two metric rows and a state; connectors show what feeds what; dashed frames name the owners; the timeline is the cadence. Design of the pipeline, not a record of a run.
Named public datasets and institutional records, each with its licence class. Sources by agreement are used only under their terms and never republished.
| Source | What it gives | Licence class |
|---|---|---|
| OpenML | Datasets, tasks and evaluation runs with standardised splits | Open, CC BY |
| Hugging Face Hub model and dataset cards | Documented models and datasets with stated training data and limitations | Licence per model or dataset |
| MLCommons benchmarks | Shared benchmarks and measurement rules for model performance | Open, Apache 2.0 |
| AI Incident Database (Responsible AI Collaborative) | Reported failures and harms of deployed AI systems | Open, attribution |
| Software Heritage | Archived source code with persistent intrinsic identifiers | Open archive, licence per repository |
| arXiv metadata | Preprints across physics, computing and quantitative biology | Metadata CC0; licence per paper |
| Zenodo (CERN) | Citable deposits of code, models, data and evaluation outputs | Open, licence per record |
Nexus Labs prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.
| Who decides | The question | Evidence this page serves |
|---|---|---|
| Journal editors and reviewers | Whether a model-based claim rests on an evaluation that excludes leakage and tests generalisation | REFORMS or TRIPOD+AI checklist, split design, external validation, code availability |
| Research funders | Whether applicants and reviewers may use generative AI, and on what disclosure terms | Funder policy, disclosure records, documented verification of generated content |
| Medicines and device regulators | Whether an AI component in a medical product is validated for its intended use | Prespecified evaluation, external validation, predetermined change control plan |
| Research integrity offices | Whether an allegation involving fabricated or generated content is substantiated | Provenance of text, data and code; version history; tool logs |
Machine learning models learn statistical regularities from data. They can be very effective where those regularities are stable and data are plentiful, for example classifying land cover from satellite imagery or emulating a climate model to explore many scenarios. They are weaker where data are sparse, where the system changes over time or where rare events matter most, which describes much of risk science.
Common failure modes are well understood: leakage, where information from the test set reaches training; distribution shift, where conditions in use differ from those in training; shortcut learning, where a model relies on a spurious cue; and overconfident outputs without calibrated uncertainty. Generative models add the risk of fluent but fabricated content, including citations and code that looks correct but is not.
Workflow effects matter as much as model accuracy. When a tool is used at several stages, small errors can accumulate, and reviewers may give automated outputs more trust than they deserve.
Evaluation on data separated by time, place or source. This is the minimum for any claim of generalisation.
Comparison with simple statistical methods and expert judgement. A complex model that does not beat a simple baseline is not adding value.
Checks that stated confidence matches observed accuracy, using uncertainty quantification.
A record of which model, version and prompts or settings were used at each step, so the workflow can be re-run or audited.
Benchmark scores rarely transfer directly to a new research setting, and a model's behaviour can change between versions. Evaluation is repeated when either the task or the model changes.
The current service level is R0: information and enquiries. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.