I have been tinkering with an idea today, but I have only taken one semester of statistics, so please bear that in mind. Thanks. First up, the question:
How do you judge multiple estimates (with standard errors included)? I don't mean estimating their mean or distribution, rather some representation of similarity. Something like R^2 (or a different goodness-of-fit measure) in linear regression models.
I have heard of meta-analysis, but that, to my knowledge, deals with the estimation, precision and requires a bit more data. I does not, AFAIK, say much about the similarity. Maybe the size of the CI of the estimate does reflect similarity in a way.
When we tested for similarity in school, we did not consider the errors, only the estimates (chi-square test, Fisher test).
My thoughts:
When you have two estimates and their standard errors, you are interested in their confidence intervals. While I know it's dangerous to judge the statistical similarity only based on these intervals instead of hypotheses tests, but what about checking the size of the overlap? Getting a ratio of something like $$ratio_{1,2} = \frac{l(overlap_{1,2})}{l(CI_1) + l(CI_2)},$$ where $l$ only returns the length of an interval. The closer to one, the better (that of course does not have to hold). I started thinking about it, because sometimes you have $n$ estimates for the same phenomenon and you would like to know the level of their similarity. For example a ratio
$$ \frac{\sum_{i,j,i<j} ratio_{i,j}}{n(n-1)/2}, $$ would be an indicator of pairwise similarity.
Just a thought. Just a brain dump from when I was walking my dog.
I Googled a bit and found two papers on the topic, but they are a bit too technical for me, they also deal with problems kind of different.
I am sorry if this is some kind of trivial topic in statistics. Or if the description of the question is not clear.
Thanks!