This is a bit of a simple question I would suspect for many users here, but for someone like me who has never covered this in my statistics classes, this is posing a bit of a problem. I want to convert forecasted scores with a regular error bar into rankings with error bars.

For example, the following individuals are to be ranked.

A [Forecasted Score: 1000, Standard deviation: 25]

B [Forecasted Score: 940, Standard deviation: 35]

C [Forecasted Score: 950, Standard Deviation: 10]

D [Forecasted Score: 910, Standard deviation: 40]

The ranking without any way of expressing how uncertain you are would be the following.

  1. A

  2. C

  3. B

  4. D

Is this done using simply by rifling through all the combinations? As in, if I chose 2 standard deviations, A could be ranked 1, 2, or 3 while B could be ranked 1, 2, 3, or 4?

This stuff was never covered in my statistics classes, and the statistics books I have handy do not cover this. Any help would be greatly appreciated.


I thought I would give a bit of background of what I am trying to do here. Suppose you are in a bar with your best friends and you are discussing a subject which can be ranked. Sports teams is popular but it can be decided clearly on the basis of who is winning and who is not. But judging songs? You're judging and ranking art. Taking a firm position on rankings might be helpful in saying that you have a ranking, but it doesn't help if someone else can make another convincing argument based on the same criteria why someone should be ranked over someone else. Sometimes it is just more helpful to describe the ranks it can have more than it does have.

This is the problem we are tackling here.

  • 1
    $\begingroup$ Your examples are notable insofar as they would plausibly induce strong dependencies among the rankings: this would force you to assess the covariances among the rankings as well as their variances. $\endgroup$ – whuber Apr 14 '17 at 22:55

How about something like this?

enter image description here

Predictive rank probabilities
rank     A     B     C     D
   1 0.882 0.076 0.019 0.023
   2 0.097 0.292 0.507 0.104
   3 0.018 0.383 0.405 0.193
   4 0.003 0.248 0.069 0.680

This is a simple simulation-based approach based on treating the forecasted means and SDs as parameters of the predictive distribution of scores. Note that in this particular case, I made the (questionable!) assumptions that the forecasted distributions of scores for each individual are Normal and independent, just because there was no other information available in the question. So, anyway, we simply simulate 10,000 draws from this distribution, compute the ranks in each simulated run, and then summarize the results as a set of predicted probabilities of ranks. These can be interpreted either as (1) For a given rank, what are the probabilities that each individual will occupy that rank? or (2) For a given individual, what are the probabilities that they will occupy ranks 1-4?

R code follows:

# simulate draws from predictive distribution of scores
draws <- mapply(rnorm,
                mean = c(1000, 940, 950, 910),
                sd = c(25, 35, 10, 40),
                MoreArgs = list(n=10000))
# tabulate ranks in each simulation
ranks <- apply(draws, 1, order, decreasing=TRUE)
# computer predicted probability
probability <- sapply(1:4, function(x) rowMeans(ranks==x))
# line plot
matplot(y=probability, x=1:4, type="o", ylim=0:1, xlab="Rank",
        col="black", lty=1, pch=LETTERS[1:4])
# display probabilities
dimnames(probability) <- list(rank=1:4, individual=LETTERS[1:4])
round(probability, 3)
| cite | improve this answer | |
  • $\begingroup$ I like it. I will give it an upvote but wait for others better than me to comment before it gets the green checkmark. $\endgroup$ – Phillip Siebold Apr 17 '17 at 3:01

Ranking non-deterministic data in a deterministic way doesn't seem very sensible to me.

Let's take for example individual $A$. His forecasted score is $1000$ and $SD$ is $25$, but that doesn't mean that it couldn't be, e.g. $900$, $1100$ or even zero. It just means that these values are outside of the "expected range", heuristically speaking.

A more extreme example why the ranking system you proposed doesn't work very well is that it would place an imaginary individual with $0$ forecasted score and $SD$ of $2000$ on top.
In particular, you can't say "Individual $X$ will be ranked highest, because the upper bound of his expected score range is highest". You can only rank the forecast and say "Individual $X$ has the highest ranked forecasted score", because the actual outcome is random.

This of course assumes that you didn't mean "range" by "standard deviation", i.e. that scores for $A$ outside of $[975,1025]$ are impossible. I also assumed that the possible range for the actual score is the same for every individual, as the scoring system wouldn't make much sense otherwise.

Either way, to resolve this problem and get a meaningful ranking, you could just rank according to the "expected/forecasted values" and denote the ranking as such, involving the standard deviation values. There is no way to determine the ranking of the actual score results just by the use of forecasts.

| cite | improve this answer | |
  • $\begingroup$ The issue I'm having is that I want to be able to select a top 10 and then be able to select those that could be in the top 10 but miss the mark. This ambiguity is important because. . . well, the subject matter (of what is being ranked) is ambiguous. $\endgroup$ – Phillip Siebold Apr 14 '17 at 15:53
  • $\begingroup$ I assume that you want to forecast the ranking of the individuals' test outcome. And every individual could be in the top 10 in the outcome, right? Otherwise the scoring system would be biased. Then it follows that you have to rank them according to the forecasted score, because the SD does not only imply that the actual score could be (much) higher, but also (much) lower. The chance is 50-50. If it weren't, you'd have to adjust the forecasted score to the side with the higher probability - otherwise it would be a deliberately bad forecast. Did I manage to clear things up? $\endgroup$ – Eldioo Apr 15 '17 at 10:22
  • $\begingroup$ Maybe. But sticking with the simple ranking as is with only the $\sigma$ showing is not as useful as showing the spread. It only makes it a less useful tool than just stating the ranking. $\endgroup$ – Phillip Siebold Apr 15 '17 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.