Randomised pilots and stepped rollouts
Where an innovation reaches sites in sequence, the order is randomised so early and late sites form a comparison, with the design and analysis registered beforehand.
GCRI · Labs
Causal inference for pilots
Causal inference separates what an intervention does from what merely accompanies it. In labs and pilots it shapes the design: randomisation where possible, credible comparisons where not, and explicit assumptions in both. It is how a pilot becomes evidence that an innovation works, for whom and under what conditions.
01Contents
Causal inference for pilots in five parts.
02Application
What causal inference and impact evaluation does here.
Causal inference and impact evaluation develop methods to determine whether and how interventions or exposures cause outcomes. They separate genuine effects from correlation. In research and evidence, methods and modelling and innovation, causal inference shows which risk reduction measures actually work. It supports allocating resources toward interventions with demonstrated effect across connected risks. The canonical description is on the STEM Commons.
Quasi-experimental programme evaluation
Difference-in-differences, regression discontinuity or synthetic control designs estimate effects where randomisation was not possible, with placebo tests to probe their assumptions.
Transporting effects from pilot to scale
Causal diagrams state which features of the pilot site an effect depends on, so planners can judge whether a new site shares them before expecting the same result.
03Monitor
Impact evaluation.
The screen a team would use to follow this method: its measures, states and decisions. Names are examples and values are illustrative.
Impact evaluation
Labs · Causal inference for pilots · Run R-7061
Parallel trends holdGCRI · Labs
Event study
Method card
- Method
- Causal inference and impact evaluation
- Design
- difference in differences
- Units
- 196
- Label
- MODELLED
Robustness checks
| Main specification | 0.06 | Yes |
| Drop early adopters | 0.26 | Yes |
| Synthetic control | 0.12 | Yes |
04Data
What the method consumes.
Datasets and records used by causal inference and impact evaluation on this platform, with the role each plays and the label its values carry. Sources are listed in the source register.
| Data | Role | Label |
|---|---|---|
| Trial and pilot outcome data | Effect estimation | OBSERVED |
| Administrative records | Outcomes and covariates for comparison groups | Published statistics |
| Registered pre-analysis plans | Fixing estimands and tests | DECLARED |
| Site characteristic surveys | Heterogeneity and transportability | REPORTED |
05Methods and evidence
How causal inference for pilots is modelled, measured and tested.
The model, its assumptions, its data and its uncertainty, in the form a scientist or engineer would check. Structures follow published methods. The values drawn are illustrative, not findings.
Model, equations and assumptions
The model structure used for causal inference for pilots, the assumptions it makes and how each is tested.
Risk of element i
Ri = f(Hi, Ei, Vi, Ci)hazard, exposure, vulnerability and coping capacityExpected annual loss
EAL = ∑k pk · Lkover events k with annual probability pkCoupled systems
xt+1 = A xt + B ut + εtA carries the couplings, u the shock, ε the error| Assumption | Why it is made | How it is tested |
|---|---|---|
| Hazard and exposure are independent within an event | Keeps the loss integral tractable | Compare with a copula fit on joint records |
| Couplings are linear near the operating point | Lets shocks propagate through A | Perturb beyond the linear range and compare |
| Vulnerability curves transfer between places | Local curves are rarely available | Validate against observed losses where they exist |
- Inputs
- versioned datasets with provenance
- Outputs
- distributions, never single numbers
- Code
- released with the model card and tests
- Reproduction
- same data, code, environment and seed
FAIR guiding principles · GO FAIR and FORCE11W3C PROV · World Wide Web ConsortiumGuidance on consistent treatment of uncertainties · IPCCCEOS Analysis Ready Data · Committee on Earth Observation Satellites
06Validation
How results are checked.
Identification assumptions are probed through placebo tests, falsification checks and sensitivity analysis. Replication across settings tests external validity.
| Check | How it is done |
|---|---|
| Balance and pre-trend checks | Comparison groups are tested for similarity before treatment |
| Placebo and falsification tests | Effects are sought where none should exist |
| Sensitivity to unobserved confounding | Analyses report how strong a hidden confounder would need to be to change the conclusion |
| Replication across sites | Effects are re-estimated in independent settings before they are generalised |
Unobserved confounding and spillovers can bias estimates. Effects found in one context may not transfer to another.
07Process
How the process runs.
Process
Causal inference for pilots: PRISMA 2020 review reporting
PRISMA 2020 is a reporting guideline for systematic reviews; the steps follow its checklist, not a mandated sequence, and no Nexus document adopts it.
01 Plan
- 01Registration and protocolRecord · The review teamThe guideline asks for registration details, access to the protocol and any amendments.
- 02Eligibility criteriaRecord · The review teamThe guideline asks for the inclusion and exclusion criteria and how studies were grouped.
02 Find and select
- 03Search: sources and strategiesStep · The review teamThe guideline asks for the information sources and the full search strategies. It does not fix which sources are used.
- 04Selection of studiesStep · The review teamThe guideline asks how studies were judged eligible, including how many reviewers screened. It does not fix the number of reviewers.
- 05Flow diagramRecordThe flow diagram shows the number of records found, the number of studies included and the reasons for exclusion.
03 Analyse
- 06Data collectionStep · The review teamThe guideline asks how data were collected from reports and which data items were sought.
- 07Risk of biasStep · The review teamThe guideline asks for the methods and tools used to assess risk of bias.
- 08SynthesisModel · The review teamThe guideline asks for the effect measures and the synthesis methods. It does not prescribe meta-analysis.
- 09Certainty of evidenceStep · The review teamThe guideline asks for the methods used to assess certainty, and reports the result per outcome. It does not name a grading scale.
04 Report
- 10Review reportRecordThe guideline helps authors report why the review was done, what they did and what they found.
ReportsOutline of an externally published reporting guideline, not a Nexus process and not a record of operation. PRISMA 2020 mainly covers reviews of intervention effects. No Nexus page names it. Every programme is at service level R0.
09Stewardship
Who maintains the method.
Economic Development stewards causal inference and impact evaluation for the Consortium and reviews how it is applied across platforms.
10Connected
Where this page leads across the Consortium.
11Questions
Questions and answers.
What are the limits of causal inference and impact evaluation in this field?
Unobserved confounding and spillovers can bias estimates. Effects found in one context may not transfer to another.
How are results checked?
Identification assumptions are probed through placebo tests, falsification checks and sensitivity analysis. Replication across settings tests external validity.
Are the values on this page real?
The data names, roles and checks are real descriptions of practice. The canvas shows a design of the method’s use, not a run; no result is reported here.
Where else is this method applied?
The canonical method page lists every platform that applies it.
ReportsContribute
Contribute evidence from this page.
Reports is the Consortium’s quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.
GCRILabsLabs EvidenceMethods and Technical
- EmailGateway GW-GCRISend it from this pageEvidence, a correction, a case, a method, a review, a translation or profile input. It arrives tagged with this page, so the editor who owns it receives it.
- ZenodoOpen depositDeposit it and submit to the communityPublish your own record on Zenodo with your ORCID iD, then submit it to the Reports community there. Curators review it; the DOI stays yours.
- ReportsGCRISee every way to contributeThe contribution types, how each is reviewed, what credit you receive, the licence and what is never accepted.
Never send personal data, sensitive infrastructure detail or anything you have no right to share. A contribution is reviewed before use; sending it confers no authorship or editorial rights, and sponsorship never buys a finding.
Labs
Bring a method, a dataset or a question.
Information and enquiries are open today. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.
Legal
Legal documents.
Maps, data and figures on this site are shown for context and general information. They are not advice, forecasts or warnings; the limits of the information are set out in the Disclaimers and Data Notices.