Research rarely produces a perfectly uniform answer. Two credible studies can examine what appears to be the same clinical question and report different effects, different levels of certainty, or conclusions that seem to point in opposite directions.
Different results may reflect differences in the clinical question, study methods, risk of bias, or precision rather than a true contradiction. Critical appraisal begins by determining whether the studies are similar enough that comparable results should reasonably have been expected.
Are the Studies Actually Asking the Same Question?
Two articles can share a topic while answering meaningfully different questions. A study of an intervention in adults with severe disease may not be directly comparable with one conducted among lower-risk patients. A treatment delivered weekly may produce a different effect from the same treatment delivered once, and an outcome measured at 30 days may not reflect what happens after a year.
Population, intervention, comparator, and outcome can differ enough that apparently conflicting studies are estimating different effects. GRADE examines whether differences in these elements can explain variation in study results (Neumann et al., 2025). Studies should be similar enough that a reasonably comparable result would have been expected before their findings are treated as contradictory.
Compare the Effects, Not the Conclusions
Article conclusions can make disagreement look sharper than it actually is. One study may describe an intervention as effective because its confidence interval excludes the null value, while another reports no statistically significant effect. A statistically significant result in one study and a nonsignificant result in another does not, by itself, demonstrate that the study effects differ.
The effect estimates and confidence intervals provide more useful information. Two estimates may point in the same direction but differ in precision. One study may suggest a relatively precise benefit, while another has a wider interval that includes both benefit and little or no effect. Cochrane recommends examining the magnitude and direction of individual study effects when interpreting heterogeneity rather than relying on statistical significance or a single heterogeneity statistic (Deeks et al., 2024). The more useful comparison is whether the observed effects fall within a reasonably similar range, not whether the abstracts use the same words.
Look for Clinical Differences That Could Explain the Results
Differences in patient populations can change the effect an intervention produces. Baseline risk, disease severity, age, comorbidities, previous treatment, and eligibility criteria may all affect how well findings transfer from one study to another.
The intervention itself may also differ. “Care coordination” could mean one follow-up phone call in one study and repeated contact from a multidisciplinary team in another. Usual care may vary just as much. An intervention tested against minimal follow-up may appear more effective than the same intervention tested where the comparison group already receives substantial support.
Outcome definitions and follow-up periods deserve the same scrutiny. Thirty-day all-cause readmission is not equivalent to disease-specific readmission at 90 days. Medication adherence measured through pharmacy records is not identical to self-report. An intervention may also show an early effect that diminishes over time or a benefit that appears only after longer follow-up.
Cochrane identifies differences in participants, interventions, outcomes, and other clinical characteristics as potential sources of heterogeneity, while GRADE considers whether variation in population, intervention, comparator, or outcomes can explain inconsistency across studies (Deeks et al., 2024; Neumann et al., 2025). When studies differ in clinically important ways, different results may reflect different effects under different circumstances rather than unexplained inconsistency.
Decide Whether Methodological Differences Change the Weight of the Evidence
Conflicting studies do not automatically deserve equal weight. A study with substantial risk of bias, considerable missing outcome data, weak outcome measurement, or important uncontrolled confounding may provide less reliable information than one with stronger methodological safeguards.
That does not justify dismissing an inconvenient result simply because limitations can be identified. Nearly every study has limitations. The issue is whether those limitations could plausibly distort the result enough to help explain the disagreement.
Cochrane defines bias as systematic deviation from the truth and notes that bias can either exaggerate or underestimate an intervention effect (Boutron et al., 2024). Risk of bias therefore belongs in the comparison regardless of which study produced the preferred result. A methodological limitation should matter whether it appears in the study you agree with or the study you do not. The Critical Appraisal Worksheet provides a consistent structure for documenting those judgments across studies.
Investigate Outliers Without Explaining Them Away
An outlying study deserves investigation, not automatic exclusion. Differences in population, intervention, comparator, outcome measurement, study methods, or data extraction may explain why its result differs from the rest. If no credible explanation emerges, the inconsistency remains part of the evidence and may warrant lower certainty in the overall effect estimate (Deeks et al., 2024; Neumann et al., 2025).
Statistical measures of heterogeneity can help identify variation, but they should not decide the interpretation by themselves. Cochrane cautions against rigid interpretation of I2 because its meaning depends on the size and direction of effects, the evidence for heterogeneity, and the clinical context (Deeks et al., 2024).
Explanations are more convincing when specified before the study results are examined than when they are developed after the inconsistency becomes visible. Post hoc analyses can generate useful hypotheses, but they provide weaker evidence that the proposed factor actually explains the disagreement (Deeks et al., 2024). Once a preferred result is already known, it becomes easier to find reasons to discount an inconvenient study. Applying the same appraisal standards across studies helps prevent that reasoning from becoming cherry-picking.
Sometimes the Evidence Really Is Inconsistent
Sometimes clinically important differences remain after population, intervention, comparator, outcomes, methodology, and precision have been examined. GRADE treats unexplained inconsistency as a reason to lower certainty in the evidence because the variation may reflect unstable effect estimates, genuine differences across settings or populations, or uncertainty that existing studies cannot yet resolve (Neumann et al., 2025).
Critical appraisal does not require forcing conflicting studies into a single answer. One study may deserve more weight because its methods are stronger or its population better matches the clinical question. In other cases, different results may reflect genuinely different clinical circumstances. When no credible explanation resolves the disagreement, uncertainty should remain visible rather than being removed through selective interpretation. The Evidence Synthesis Writing Guide & Worksheet offers a structured way to carry those distinctions into the written synthesis.
Comparing conflicting studies should begin with the studies themselves rather than the conclusion you hope to reach. The conclusion should follow from the evidence that survives the comparison, not from which study produced the answer you wanted.
References
Boutron, I., Page, M. J., Higgins, J. P. T., Altman, D. G., Lundh, A., & Hróbjartsson, A. (2024). Considering bias and conflicts of interest among the included studies. In J. P. T. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page, & V. A. Welch (Eds.), Cochrane handbook for systematic reviews of interventions (Version 6.5). Cochrane. https://training.cochrane.org/handbook/current/chapter-07
Deeks, J. J., Higgins, J. P. T., Altman, D. G., McKenzie, J. E., & Veroniki, A. A. (2024). Analysing data and undertaking meta-analyses. In J. P. T. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page, & V. A. Welch (Eds.), Cochrane handbook for systematic reviews of interventions (Version 6.5). Cochrane. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-10
Neumann, I., Sousa-Pinto, B., Meerpohl, J., Dahm, P., Brennan, S., Alonso, P., Akl, E., Langendam, M., Guyatt, G., & Schünemann, H. J. (2025). Inconsistency. In I. Neumann & H. J. Schünemann (Eds.), The GRADE book (Version 1.0). GRADE Working Group. https://book.gradepro.org/guideline/inconsistency