0
$\begingroup$

Often test takers are ranked based on their test scores (e.g., people taking a civil service exam) and often those test scores are rounded before being ranked, producing many ties). How might I go about calculating a confidence interval around such ranks (in order to give the decision makers who use the ranked list an estimate of the degree of confidence the user should have in the order of the rankings). I would like to describe and convey to users some information about whether a different of k ranks is likely to reflect a true difference or is likely to arise often based on chance. I presume the confidence interval will depend on the numeric score underlying the ranked list. Is there a general solution, perhaps based on the mean, s.d., and number of people tested? Assume the underlying distribution is normal.

$\endgroup$
  • 2
    $\begingroup$ Could you clarify what such a confidence interval would represent? One can think of several distinct interpretations of your question, depending on what the objective of the analysis is: to find an interval for a particular individual or intervals for all individuals and how exactly such intervals ought to be interpreted, for instance. Would the analysis of confidence intervals for percentiles be appropriate for your problem? $\endgroup$ – whuber May 20 at 12:52
  • $\begingroup$ @whuber. I have clarified the question. Since the underlying score distribution may be assumed to be normal, the links you provide are not maximally useful (as they are distribution free). $\endgroup$ – Joel W. May 20 at 15:30
  • 1
    $\begingroup$ Since the question focuses on ranks, it appears likely that you will gain nothing by making such a distributional assumption. Indeed, the discrete nature of test scores produces a likelihood of ties in ranks--perhaps a high likelihood of many ties--which would never occur with a truly Normal distribution. Beware, then, that your distributional assumption could produce incorrect results. A more serious problem is that you don't have the necessary information: on the basis of just one test, how could you possibly attribute differences in test scores to individuals instead of pure chance? $\endgroup$ – whuber May 20 at 15:36
  • $\begingroup$ @whuber. I did say in the question that I thought the answer would depend on the score and the number of people tested. The advantage of assuming a normal curve are: (1) it reflects reality, and (2) it will help to more accurately reflect the non-uniform distribution of scores. You are correct about ties, especially since the ranks are often based on rounded test scores (depending on the policy of the testing organization). $\endgroup$ – Joel W. May 20 at 15:58
  • $\begingroup$ What is most important is whether the Normal model reflects the aspects of the reality that matter to the question. Although it might be a beautiful approximation to the distribution of test scores for routine purposes, like comparing means or variances, your special circumstances a Normal approximation might depart hugely from reality in terms of quantities that really matter, such as the likelihood of ties. This is one huge advantage nonparametric methods have over parametric ones: they are less likely to go wrong when your assumptions aren't quite correct. $\endgroup$ – whuber May 20 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.