2
$\begingroup$

I apologize in advance for the somewhat long description. Which statistic is the following (if any)?

Let's say we have 100 colleges, each of which has conducted a survey of its students about satisfaction. So each college has an average score from its survey responses, and we then rank the 100 colleges based on their scores. But each survey has a different number of respondents. So we calculate a margin-of-error for each survey. So now we have 100 scores, and their 100 accompanying margins-of-error.

For example, let’s say Western University has a score of 8.3 out of 10 and a margin-of-error of 1.5. So that means we know with 95% confidence that Western University's true score is somewhere between 6.8 and 9.8.

And now we want to know generally how "overlapping" our list of 100 ranked scores are, based on these confidence intervals. Is our overall list relatively "fuzzy" or "clear"?

So we run some random trials. We generate a random number from 6.8 to 9.8 for Western University. We do the same for each of the other 100 colleges, using its appropriate confidence interval. Then we re-rank them. Then we subtract the absolute distance change in rank for each college.

For example, let’s say Western University was originally ranked #50 out of 100 and then after our shuffling it became ranked #37. So Western University changed 13 places in that trial.

And we run this kind of randomized trial, again and again, thousands of times, and take the average change in rank of all colleges for all trials.

So a perfectly stable, "never-overlapping" set (with 0 for margin-of-error for all colleges) would have an average change in rank from thousands of trials of 0.

And a perfectly "overlapping" set (with a margin-of-error of 50+ for all colleges) would have an average change in rank of 33.

For any actual set, we would take the average change in rank, from 0 to 33 in this example, and divide it into the max (33 in this example) to normalize. So let’s say our average was 16.5 of 33 so we divide and our final statistic is 0.50.

This would give us a sense of how "overlapping" the scores were. A score of 0.99 would be a very "hazy" set of scores, while a score of 0.01 would be a relatively "clear" set of scores.

Does something like this exist?

$\endgroup$
1
$\begingroup$

Sounds to me like the average Spearman correlation between the observed scores and randomly generated scores (based on observed scores and margin of errors).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.