Confidence interval overlap - judging the similarity of estimates

Question

I have been tinkering with an idea today, but I have only taken one semester of statistics, so please bear that in mind. Thanks. First up, the question:

How do you judge multiple estimates (with standard errors included)? I don't mean estimating their mean or distribution, rather some representation of similarity. Something like R^2 (or a different goodness-of-fit measure) in linear regression models.

I have heard of meta-analysis, but that, to my knowledge, deals with the estimation, precision and requires a bit more data. I does not, AFAIK, say much about the similarity. Maybe the size of the CI of the estimate does reflect similarity in a way.

When we tested for similarity in school, we did not consider the errors, only the estimates (chi-square test, Fisher test).

My thoughts:

When you have two estimates and their standard errors, you are interested in their confidence intervals. While I know it's dangerous to judge the statistical similarity only based on these intervals instead of hypotheses tests, but what about checking the size of the overlap? Getting a ratio of something like $$ratio_{1,2} = \frac{l(overlap_{1,2})}{l(CI_1) + l(CI_2)},$$ where $l$ only returns the length of an interval. The closer to one, the better (that of course does not have to hold). I started thinking about it, because sometimes you have $n$ estimates for the same phenomenon and you would like to know the level of their similarity. For example a ratio

$$ \frac{\sum_{i,j,i<j} ratio_{i,j}}{n(n-1)/2}, $$ would be an indicator of pairwise similarity.

Just a thought. Just a brain dump from when I was walking my dog.

I Googled a bit and found two papers on the topic, but they are a bit too technical for me, they also deal with problems kind of different.

I am sorry if this is some kind of trivial topic in statistics. Or if the description of the question is not clear.

Thanks!

Answer 1

Regardless of what you do, you will be making assumptions around the direct comparability of studies that took measures at different times, used different predictors (or, different definitions of the "same" predictor), and even measured the "same" predictor in different ways. The effect of these differences can be expected to alter the estimate and its associated confidence interval between studies. Sample size also has an effect here, because all things being equal the study with the larger sample size will have a smaller confidence interval around the estimate because increasing sample size decreases variance.

It would be good to have the raw data for each study. I'm assuming that is not possible so you will only be able to work from the summary statistics, and that suitable summary statistics are published. This may cut down the number of tests you can do.

Assuming you have more than two studies, you could do an ANOVA on the estimate with study as the group. This will tell you whether the studies significantly differ in their estimates. If the software won't accept summary statistics as input, you will need to generate a mock sample of correct size using the summary statistics in the study, for each study.

Another possible way forward is meta-analysis. If you look at the forest plot output from a meta-analysis, this shows the level of overlap of the confidence intervals between the studies, as well as drawing a conclusion about where the true estimate lies, which of course is influenced by each study that is included in the meta-analysis.

Otherwise, this previous question or the solution proposed here are other ways forward.