Skip to content

GCRI · Labs

Reproducibility and negative results

Science corrects itself only when failures are recorded: replications that did not reproduce, experiments that found nothing, errors found after publication. Many of these records never appear, so the published literature overstates effects and certainty. This page sets out how irreproducibility arises, how replication and correction work, which records exist, and what editors, integrity offices and funders rely on when they correct or retract.

01رجسٹر

Reproducibility and negative results on the record.

02The question

Reproducibility and negative results: how the risk develops.

Failure records and corrections. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.

طریقۂ کار

Publication bias filters the record

Significant, novel results are more likely to be written up, accepted and cited than null ones. The literature then shows a biased sample of what was studied, and meta-analyses inherit the bias.

طریقۂ کار

Errors propagate through citation

A flawed or retracted paper keeps being cited, often without mention of its status. Reviews, guidelines and later studies that build on it carry the error forward, sometimes long after the notice.

طریقۂ کار

Reproduction fails for ordinary reasons

Missing code, undocumented software versions, unshared reagents, unclear methods and lost raw data stop others from repeating an analysis or experiment, even when the original work was honest.

طریقۂ کار

Correction is slow and uneven

Investigations take time, journals differ in how they flag problems, and retraction notices can be vague. Readers may not learn that a paper was corrected, and databases may not carry the notice forward.

03نگرانی

Protocol and replication board.

وہ اسکرین جو ایک ٹیم اس موضوع کی پیروی کے لیے استعمال کرے گی: اس کے پیمانے، حالتیں اور فیصلے۔ نام مثالیں ہیں اور اقدار توضیحی ہیں۔

Protocol and replication board

Reproducibility and negative results · Labs · Quarter 1

Two replications runningGCRI · Labs

Protocols

PR-594 sensor driftField experimentہاںمنظور شدہRunning
PR-714 AI triageRandomisedہاںConditionsSetup
PR-551 message recallSurvey experimentہاںمنظور شدہComplete
PR-896 heat exposureگروہNoPendingمسودہ

Replication R-20

Original effect
0.3 SD
Replication effect
0.3 SD
Verdict
Smaller, same sign
شائع شدہ
with data and code

Testbed availability

Protocol and replication board: the screen’s structure, terms and states for Reproducibility and negative results.Example names and illustrative values; a design, not a record or a live system.
04In the five domains

Negative results matter in every domain.

Negative results are published in the Labs Evidence series. The National Academies’ report (2019) distinguishes reproducibility from replicability; DORA (2012) supports assessing research on its merits.

  • Negative results.
  • Replication attempts.
05Methods and evidence

How reproducibility and negative results is modelled, measured and tested.

The model, its assumptions, its data and its uncertainty, in the form a scientist or engineer would check. Structures follow published methods. The values drawn are illustrative, not findings.

Model, equations and assumptions

The model structure used for reproducibility and negative results, the assumptions it makes and how each is tested.

Risk of element iRi = f(Hi, Ei, Vi, Ci)hazard, exposure, vulnerability and coping capacity
Expected annual lossEAL = ∑k pk · Lkover events k with annual probability pk
Coupled systemsxt+1 = A xt + B ut + εtA carries the couplings, u the shock, ε the error
AssumptionWhy it is madeHow it is tested
Hazard and exposure are independent within an eventKeeps the loss integral tractableCompare with a copula fit on joint records
Couplings are linear near the operating pointLets shocks propagate through APerturb beyond the linear range and compare
Vulnerability curves transfer between placesLocal curves are rarely availableValidate against observed losses where they exist
Inputs
versioned datasets with provenance
Outputs
distributions, never single numbers
کوڈ
released with the model card and tests
Reproduction
same data, code, environment and seed
Standard risk and state-space formulations; the assumptions listed are typical of the method, stated for review, not results.
FAIR guiding principles · GO FAIR and FORCE11W3C PROV · World Wide Web ConsortiumGuidance on consistent treatment of uncertainties · IPCCCEOS Analysis Ready Data · Committee on Earth Observation Satellites
06اشاریے

What is measured, and how well.

Each indicator carries its unit, its cadence, the kind of source behind it and the evidence label a value would carry. No values are shown: a programme supplies them under its own authority.

IndicatorUnitCadenceSource typeلیبل
Computational reproducibilitypercent of sampled studies regenerated from shared code and dataper auditReproducibility auditsمشاہدہ شدہ
Replication outcomeproportion of replications meeting the prespecified criterionper programmeCoordinated replication projectsمشاہدہ شدہ
Retraction noticescount by field and reasonannualRetraction Watch database via CrossrefPublished statistics
Time from concern to correctionmonthsper caseJournal and integrity office recordsرپورٹ شدہ
Post-retraction citationscitations per retracted articleannualCitation data linked to retraction recordsمشاہدہ شدہ
Null and negative results depositedcount of recordsannualRepositories and registriesرپورٹ شدہ
Data available on requestpercent of articles whose data could be obtainedper auditData request auditsمشاہدہ شدہ
07ماخذات

Where the evidence comes from.

Named public datasets and institutional records, each with its licence class. Sources by agreement are used only under their terms and never republished.

ماخذWhat it givesLicence class
Retraction Watch database (Crossref)Retractions, corrections and expressions of concern with reasonsOpen, attribution
Crossref metadata and Crossmark updatesLinks from articles to their corrections and retractionsOpen metadata, no restrictions
OpenAlexWorks, citations, authors and institutions across disciplinesOpen, CC0
FORRT Replication DatabaseReplication attempts and outcomes, mostly in the social and behavioural sciencesOpen, attribution
OSF projects of coordinated replication programmesProtocols, data and code of large-scale replicationsOpen, licence per project
PubPeerPost-publication comments identifying possible problems in papersFree access, terms of use
Zenodo (CERN)Citable deposits for null results, data and codeOpen, licence per record
08Decisions

Who decides, and what the evidence serves.

Labs prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.

کون فیصلہ کرتا ہےThe questionEvidence this page serves
Journal editors and publishersWhether to correct, retract or issue an expression of concernEvidence of error or misconduct, author response, COPE guidance, reproduction attempts
Research integrity officesWhether an allegation is substantiated and what followsRaw data, records, code, interviews and an independent analysis
Research fundersWhether to fund replications and require data and code sharingReproducibility audit results, replication evidence by field, cost of sharing
Guideline developers and systematic reviewersWhether to rely on a study that has been questioned or correctedRetraction and correction status, replication evidence, risk-of-bias assessment

Outputs this topic produces

  • Replication protocols registered before data collection, with success criteria fixed
  • Reproducibility records listing code, data, environment and the outcome of each rerun
  • Deposits of null and negative results with full methods
  • Correction logs that link each notice to the works affected
09Core questions

What experts ask about reproducibility.

  • Can an independent team re-run the analysis from the published data and code?
  • Has the finding been replicated with new data, in a new setting or by a different team?
  • Which null or negative results exist, and where are they recorded?
  • When an error is found, how is it corrected and how are users of the result told?
  • Which parts of a workflow, such as software versions or random seeds, could change the answer?
10Mechanisms and drivers

Why results fail to reproduce.

Many published findings in several disciplines have proved difficult to replicate. The causes are well understood: small samples, flexible analysis, selective reporting, undocumented data processing and incentives that favour novel positive findings. In computational work, results can also depend on software versions, hardware, random seeds and data that change after publication.

Negative results are under-reported because they are harder to publish and seem less interesting. This distorts the evidence base: if only successful trials of an intervention are visible, the intervention looks more effective than it is. Systematic reviews and models built on that evidence inherit the bias.

Corrections matter for the same reason. An error in a widely used dataset or model can propagate into many downstream analyses before it is found.

11Methods and data

Practices that support reproducibility.

  1. 01

    Open data and code

    Sharing the materials needed to re-run an analysis, within privacy and licensing limits. Some data cannot be shared and must be described instead.

  2. 02

    Computational environments

    Recording software versions and environments, a practice developed with reproducible modelling. Long-term preservation remains hard.

  3. 03

    Registered reports

    Peer review of the protocol before results exist, so publication does not depend on the outcome.

  4. 04

    Correction records

    Dated, visible corrections linked to the original result, rather than silent replacement.

A failed replication does not by itself show the original was wrong; differences in setting, sample or method must be examined before conclusions are drawn.

12طریقے

How it is studied, and what the methods cannot say.

  1. Probability and statisticsSample size and power planning · Meta-analysis with bias assessmentApplied here
  2. Uncertainty quantification and model validationMeasurement uncertainty budgets · Sensitivity analysis of simulation modelsApplied here
  3. Causal inference and impact evaluationRandomised pilots and stepped rollouts · Quasi-experimental programme evaluationApplied here
  4. Artificial intelligence and machine learningAssisted screening for systematic reviews · Surrogate models to guide experimentsApplied here
13Where this page’s references

Where this page’s references lead.

وہ ادارے اور دستاویزات جن کے تناظر میں یہ صفحہ پڑھا جاتا ہے۔ حیثیت اور ایڈیشن بدلتے رہتے ہیں؛ ہر لنک جاری کرنے والے ادارے تک جاتا ہے۔

14سوالات

سوالات و جوابات۔

What is the difference between reproducibility and replicability?
In the usage of the US National Academies, reproducibility means obtaining the same results from the same data and code, and replicability means obtaining consistent results from new data. Usage varies by field, so definitions should be stated.
Why publish negative results?
They show what did not work, prevent wasted repetition and correct the bias that makes effects look larger than they are. A well-designed study with a null result is a finding.
Does retraction always mean misconduct?
No. Papers are also retracted for honest error, irreproducible results or publisher problems. COPE guidance asks that each notice states the reason.
Can a retracted paper still be cited?
It can be cited as retracted, for example in studies of the literature. Citing it as valid evidence is an error that reviewers and editors should catch.
Will Labs publish a study that found nothing?
Yes. A well-conducted study with a null result is designed to be recorded with the same detail as a positive one.
How do I report an error in a Labs result?
Use the correction routes on the Labs trust page. The acknowledgement gives a reference and names the next step.
Are corrected results removed?
No. The original stays visible with a dated correction, so anyone who used it can see what changed.
رپورٹیںحصہ ڈالیں

اس صفحے سے شواہد فراہم کریں۔

Reports کنسورشیم کی سہ ماہی اشاعت ہے۔ اس موضوع کے بارے میں آپ کا علم اگلے شمارے میں شواہد، کیس یا تصحیح بن سکتا ہے، جو آپ کے نام سے منسوب ہوگا اور اگر یہ آپ کا اپنا کام ہو تو CC BY 4.0 کے تحت شائع ہوگا۔

GCRIتجربہ گاہیںتجربہ گاہوں کے شواہدطریقے اور تکنیکی امور
تجربہ گاہیں

Bring a method, a dataset or a question.

معلومات اور استفسارات آج کھلے ہیں۔ استفسار کو ایک حوالہ نمبر ملتا ہے اور اگلے مرحلے کی نشاندہی ہوتی ہے؛ درخواستیں، ادائیگیاں، کمپیوٹنگ کی تخصیص اور رجسٹریشن صرف اس وقت کھلتی ہیں جب ان کی ٹیم، شرائط اور جائزے کی استعداد موجود ہو۔