Skip to content

GCRI · Labs

재현성과 부정적 결과

Science corrects itself only when failures are recorded: replications that did not reproduce, experiments that found nothing, errors found after publication. Many of these records never appear, so the published literature overstates effects and certainty. This page sets out how irreproducibility arises, how replication and correction work, which records exist, and what editors, integrity offices and funders rely on when they correct or retract.

문의하기 주제로 돌아가기
01등록

Reproducibility and negative results on the record.

자료집Labs · topic
  1. 01질문메커니즘 4개
  2. 02분야육각형 필드 판
  3. 03지표7
  4. 04출처7
  5. 05결정 권한 보유자4
  6. 06방법4
02질문

Reproducibility and negative results: how the risk develops.

Failure records and corrections. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.

메커니즘

Publication bias filters the record

Significant, novel results are more likely to be written up, accepted and cited than null ones. The literature then shows a biased sample of what was studied, and meta-analyses inherit the bias.

메커니즘

Errors propagate through citation

A flawed or retracted paper keeps being cited, often without mention of its status. Reviews, guidelines and later studies that build on it carry the error forward, sometimes long after the notice.

메커니즘

Reproduction fails for ordinary reasons

Missing code, undocumented software versions, unshared reagents, unclear methods and lost raw data stop others from repeating an analysis or experiment, even when the original work was honest.

메커니즘

Correction is slow and uneven

Investigations take time, journals differ in how they flag problems, and retraction notices can be vague. Readers may not learn that a paper was corrected, and databases may not carry the notice forward.

03모니터링

프로토콜 및 재현 보드.

팀이 이 주제를 따라가는 데 사용할 화면으로, 그 지표, 상태, 결정을 보여 줍니다. 이름은 예시이며 값은 설명용입니다.

프로토콜 및 재현 보드

Reproducibility and negative results · Labs · Quarter 1

반복 실험 두 건 실행 중GCRI · Labs

프로토콜

PR-594 sensor drift현장 실험예승인됨실행 중
PR-714 AI triage무작위화예조건설정
PR-551 message recall설문 실험예승인됨완료
PR-896 heat exposure코호트아니요대기 중초안

Replication R-20

원래 효과
0.3 SD
재현 효과
0.3 SD
판정
더 작고, 같은 부호
발행됨
데이터와 코드와 함께

테스트베드 가용성

Protocol and replication board: the screen’s structure, terms and states for Reproducibility and negative results.예시 이름과 예시 값으로, 실제 기록이나 운영 중인 시스템이 아닌 설계안입니다.
04In the five domains

Negative results matter in every domain.

Negative results are published in the Labs 증거 시리즈. The National Academies’ report (2019) distinguishes reproducibility from replicability; DORA (2012) supports assessing research on its merits.

  • Negative results.
  • Replication attempts.
05방법과 근거

How reproducibility and negative results is modelled, measured and tested.

모형, 그 가정, 데이터, 불확실성을 과학자나 엔지니어가 확인할 형태로 제시합니다. 구조는 발행된 방법을 따릅니다. 제시된 값은 예시이며 결과가 아닙니다.

모형, 방정식 및 가정

The model structure used for reproducibility and negative results, the assumptions it makes and how each is tested.

요소 i의 리스크Ri = f(Hi, Ei, Vi, Ci)재해, 노출, 취약성, 대처 역량
연간 기대 손실EAL = ∑k pk · Lk연간 확률이 pk
결합 시스템xt+1 = A xt + B ut + εtA carries the couplings, u the shock, ε the error
가정만드는 이유시험 방법
한 사건 안에서 재해와 노출은 서로 독립적임손실 적분을 다루기 쉽게 유지함결합 기록에 적합한 코퓰러와 비교
결합은 운전점 근처에서 선형이다충격이 A를 통해 전파되게 함선형 범위를 넘어 섭동을 주고 비교하십시오
취약성 곡선은 장소 간에 옮겨 쓸 수 있다지역 곡선은 거의 없다관측된 손실이 있는 곳에서는 이에 대해 검증
투입 자료
출처 정보가 있는 버전 관리 데이터세트
산출물
분포이며 결코 단일 숫자가 아님
코드
모델 카드 및 시험과 함께 공개
재현
같은 데이터, 코드, 환경과 시드
표준 리스크 및 상태공간 정식화로, 나열된 가정은 해당 방법에 일반적인 것이며 검토를 위해 명시한 것으로 결과가 아닙니다.
FAIR 지침 원칙 · GO FAIR와 FORCE11W3C PROV · 월드 와이드 웹 컨소시엄불확실성의 일관된 처리에 관한 지침 · IPCCCEOS 분석 준비 데이터(Analysis Ready Data) · 지구관측위성위원회
06지표

측정되는 것과 그 정확도.

각 지표에는 단위, 주기, 그 뒤의 출처 종류, 값에 붙을 근거 표시가 있습니다. 값은 표시하지 않습니다. 값은 프로그램이 자체 권한으로 제공합니다.

지표단위주기출처 유형라벨
Computational reproducibilitypercent of sampled studies regenerated from shared code and dataper auditReproducibility audits관측
Replication outcomeproportion of replications meeting the prespecified criterionper programmeCoordinated replication projects관측
Retraction noticescount by field and reason연간Retraction Watch database via Crossref공표 통계
Time from concern to correction개월per caseJournal and integrity office records보고됨
Post-retraction citationscitations per retracted article연간Citation data linked to retraction records관측
Null and negative results depositedcount of records연간Repositories and registries보고됨
Data available on requestpercent of articles whose data could be obtainedper auditData request audits관측
07출처

근거의 출처.

이름이 명시된 공개 데이터셋과 기관 기록으로, 각각 라이선스 등급이 있습니다. 협약에 따른 출처는 그 조건에 따라서만 사용하며 결코 재게시하지 않습니다.

출처제공하는 것라이선스 등급
Retraction Watch database (Crossref)Retractions, corrections and expressions of concern with reasons공개, 출처 표시
Crossref metadata and Crossmark updatesLinks from articles to their corrections and retractionsOpen metadata, no restrictions
OpenAlexWorks, citations, authors and institutions across disciplines공개, CC0
FORRT Replication DatabaseReplication attempts and outcomes, mostly in the social and behavioural sciences공개, 출처 표시
OSF projects of coordinated replication programmesProtocols, data and code of large-scale replicationsOpen, licence per project
PubPeerPost-publication comments identifying possible problems in papersFree access, terms of use
Zenodo (CERN)Citable deposits for null results, data and codeOpen, licence per record
08결정

누가 결정하며 근거는 무엇에 쓰이는지.

랩 prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.

결정 주체질문이 페이지가 제공하는 근거
Journal editors and publishersWhether to correct, retract or issue an expression of concernEvidence of error or misconduct, author response, COPE guidance, reproduction attempts
Research integrity officesWhether an allegation is substantiated and what followsRaw data, records, code, interviews and an independent analysis
연구 재원 제공 기관Whether to fund replications and require data and code sharingReproducibility audit results, replication evidence by field, cost of sharing
Guideline developers and systematic reviewersWhether to rely on a study that has been questioned or correctedRetraction and correction status, replication evidence, risk-of-bias assessment

이 주제가 생산하는 산출물

  • Replication protocols registered before data collection, with success criteria fixed
  • Reproducibility records listing code, data, environment and the outcome of each rerun
  • Deposits of null and negative results with full methods
  • Correction logs that link each notice to the works affected
09핵심 질문

What experts ask about reproducibility.

  • Can an independent team re-run the analysis from the published data and code?
  • Has the finding been replicated with new data, in a new setting or by a different team?
  • Which null or negative results exist, and where are they recorded?
  • When an error is found, how is it corrected and how are users of the result told?
  • Which parts of a workflow, such as software versions or random seeds, could change the answer?
10메커니즘과 동인

Why results fail to reproduce.

Many published findings in several disciplines have proved difficult to replicate. The causes are well understood: small samples, flexible analysis, selective reporting, undocumented data processing and incentives that favour novel positive findings. In computational work, results can also depend on software versions, hardware, random seeds and data that change after publication.

Negative results are under-reported because they are harder to publish and seem less interesting. This distorts the evidence base: if only successful trials of an intervention are visible, the intervention looks more effective than it is. Systematic reviews and models built on that evidence inherit the bias.

Corrections matter for the same reason. An error in a widely used dataset or model can propagate into many downstream analyses before it is found.

11방법과 데이터

Practices that support reproducibility.

  1. 01

    Open data and code

    Sharing the materials needed to re-run an analysis, within privacy and licensing limits. Some data cannot be shared and must be described instead.

  2. 02

    Computational environments

    Recording software versions and environments, a practice developed with 재현 가능한 모델링. Long-term preservation remains hard.

  3. 03

    Registered reports

    Peer review of the protocol before results exist, so publication does not depend on the outcome.

  4. 04

    Correction records

    Dated, visible corrections linked to the original result, rather than silent replacement.

A failed replication does not by itself show the original was wrong; differences in setting, sample or method must be examined before conclusions are drawn.

12방법

연구되는 방식과 방법이 말할 수 없는 것.

  1. 확률과 통계표본 크기와 검정력 계획 · 편향 평가를 포함한 메타분석이곳에서의 적용
    정본 · STEM Commons
  2. 불확실성 정량화와 모델 검증측정 불확실성 예산 · 시뮬레이션 모델의 민감도 분석이곳에서의 적용
    정본 · STEM Commons
  3. 인과 추론과 영향 평가무작위 시범 사업과 단계적 도입 · 준실험적 프로그램 평가이곳에서의 적용
    정본 · STEM Commons
  4. 인공지능과 머신러닝체계적 문헌 고찰을 위한 보조 선별 · 실험을 이끄는 대리 모델이곳에서의 적용
    정본 · STEM Commons
13Where this page’s references

Where this page’s references lead.

이 페이지를 읽을 때 기준이 되는 기구와 문서. 상태와 판은 바뀌며, 각 링크는 발행 기구로 연결됩니다.

14질문

질문과 답변.

What is the difference between reproducibility and replicability?
In the usage of the US National Academies, reproducibility means obtaining the same results from the same data and code, and replicability means obtaining consistent results from new data. Usage varies by field, so definitions should be stated.
Why publish negative results?
They show what did not work, prevent wasted repetition and correct the bias that makes effects look larger than they are. A well-designed study with a null result is a finding.
Does retraction always mean misconduct?
No. Papers are also retracted for honest error, irreproducible results or publisher problems. COPE guidance asks that each notice states the reason.
Can a retracted paper still be cited?
It can be cited as retracted, for example in studies of the literature. Citing it as valid evidence is an error that reviewers and editors should catch.
Will Labs publish a study that found nothing?
Yes. A well-conducted study with a null result is designed to be recorded with the same detail as a positive one.
How do I report an error in a Labs result?
Use the correction routes on the Labs trust page으로 보내 주십시오. 접수 확인에는 참조 번호와 다음 단계가 안내됩니다.
Are corrected results removed?
No. The original stays visible with a dated correction, so anyone who used it can see what changed.
보고서기여하기

이 페이지에서 증거를 기여하십시오.

보고서 is the Consortium’s quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.

GCRI랩Labs 근거방법 및 기술

개인정보, 민감한 인프라 세부 정보, 공유할 권리가 없는 자료는 절대 보내지 마십시오. 기여물은 사용 전에 검토됩니다. 기여물을 보낸다고 해서 저작자 지위나 편집 권한이 부여되지 않으며, 후원으로 조사 결과를 얻을 수 없습니다.

랩

방법, 데이터세트 또는 질문을 가져오십시오.

정보와 문의는 오늘부터 열려 있습니다. 문의에는 참조 번호가 부여되고 다음 단계가 안내됩니다. 신청, 결제, 컴퓨팅 할당, 등록은 해당 팀, 약관, 심사 역량이 갖추어졌을 때만 열립니다.

방법 제안 모든 연락 경로 보기
법률

이 사이트의 지도, 데이터와 그림은 맥락과 일반 정보로 제시됩니다. 조언, 예보 또는 경보가 아니며, 정보의 한계는 면책 고지와 데이터 고지에 나와 있습니다.