0
$\begingroup$

I'm reading a meta-analysis that compared the risk of an adverse outcome in the intervention and control group. The meta-analysis yielded a summary effect size (risk difference) of +3.9%, with a 95% CI -1% to +9%. The authors concluded that this confidence interval 'exclude >1% reduction in absolute risk' because the lower bound of the CI is -1%. As I understand it, this interpretation of the 95% CI of the summary effect is incorrect - the authors should make such a claim based on the prediction interval.

To illustrate the point, I recreated the meta-analysis and found that the prediction interval was considerably wider than 95% CI of the summary effect, and it did include values less than -1% (extends as low as -7.4%). However, I am having difficulty interpreting this prediction interval because the calculated T2 is 0. (Image attached)

  1. Shouldn't the prediction interval match the confidence interval when T2 is 0?
  2. Is it fair to argue that an effect size of less than -1% is possible based on the prediction interval even though the results suggests there is no between study heterogeneity? This confuses me because, if there is no heterogeneity, then I would think that implies there is one true underlying effect size and it wouldn't make sense for me to suggest a distribution of effect sizes that includes values < -1%.

enter image description here

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Re (1): prediction intervals should always be wider than corresponding confidence intervals. $\endgroup$
    – whuber ♦
    Commented Jul 25, 2023 at 17:56
  • 1
    $\begingroup$ "In the absence of between-study heterogeneity, the prediction interval coincides with the respective CI. However, in case of heterogeneity, a prediction interval covers a wider range than a CI." bmjopen.bmj.com/content/bmjopen/6/7/e010247.full.pdf $\endgroup$
    – Clueless
    Commented Jul 25, 2023 at 18:22
  • 1
    $\begingroup$ Thank you. The authors adopt an unconventional sense of the term "prediction interval" because they use it to refer to all "studies that might be conducted in the future." It is, in effect, a tolerance interval for the distribution of "true effects." It is close to the Bayesian meaning of a "prediction interval," but its direct comparison to confidence intervals--which are not a Bayesian concept--is then strange. $\endgroup$
    – whuber ♦
    Commented Jul 25, 2023 at 18:26
  • $\begingroup$ Thanks for your response. I guess my question then is, why are prediction intervals always wider than corresponding confidence intervals even when heterogeneity is absent? $\endgroup$
    – Clueless
    Commented Jul 25, 2023 at 18:40
  • $\begingroup$ I cannot make any sense of that (not your fault) because the admixture of Bayesian and Frequentist concepts implies that the two kinds of intervals (in these authors' apparent senses) can't even be compared. I suspect the authors might have adopted some idiosyncratic definition of "confidence interval," too, but I haven't read the article to check. That's key: we need to know precisely what they mean by each type of interval and what assumptions (such as prior distributions) go into their construction before we can even understand such a statement correctly. $\endgroup$
    – whuber ♦
    Commented Jul 25, 2023 at 18:58

1 Answer 1

Reset to default
1
$\begingroup$
  1. The calculated 95% prediction interval (95% PI) will not match the 95% confidence interval (95%) because it is (most likely) constructed with a t distribution, allowing for more uncertainty. Frequentist 95% PIs are typically calculated using a t distribution.

When the estimated $\tau^2$ = 0, we only expect 95% PI to match 95% CI for a large number of studies.

There is no battle between frequentists and Bayesians. 95% PIs are probabilistic. "A prediction interval is an estimate of an interval in which a future observation will fall, with a certain probability, given what has already been observed." (Source: https://en.wikipedia.org/wiki/Prediction_interval). I would add "given the uncertainty of the data/evidence and what has already been observed".

  1. Based on the available data and uncertainty, there is a 95% probability that the true (central) risk differences to be estimated in future studies or settings lie between -0.074 and 0.152. It is important to note that your statement "there is no heterogeneity" is suboptimal. The method used (not described), and the four available studies do not indicate any detectable between-study variance. It can be large but could not be detected - given the small number of studies. This is not the same as inferring there is "no statistical heterogeneity between studies".

Based on the graph you described, there is a fair chance that the true risk difference across studies, populations or different settings is below -1%

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.