Ranking by score with confidence intervals

Question

I am using simulation to compute a unique score for every college basketball team, and ranking these teams based on that score.

I am sensitive to the fact that the score sometimes differs by a tiny amount (1 part in 1,000), which is unlikely to be meaningfully different. Therefore, I am planning to use permutation to generate confidence intervals for these scores.

I planned to use the intersection of confidence intervals to determine if two teams had truly different rankings. For example:

1 (Team A)    0.1234 (CI 0.1232-0.1236)
2 (Team B)    0.1232 (CI 0.1228-0.1236)
3 (Team C)    0.1229 (CI 0.1227-0.1231)

In this example, however, I am uncertain about whether 2 teams should share the same rank, whether all 3 should, or whether they should all be ranked differently. Arguments for each:

Why to rank all the same:

Team A's confidence intervals overlap with those of Team B, and Team B's confidence intervals overlap with Team C. I think this can be dismissed because Team A's confidence intervals don't overlap with Team C, so it is clear that A and C should have different ranks, regardless of B's rank.

Why to rank A and B #1, and C #3:

Team A and B's confidence intervals overlap. Team A's and Team C's do not. We will define our ranking procedure to state that each rank terminates when the lower confidence bound of the team with the highest mean rank no longer overlaps with that of a subsequent team. In this case, this would cause A and B to tie for a rank of #1, and C to fall out into the next rank (#3).

Why to rank all separately:

Team A and B overlap, and team B and C overlap. If Team A and Team B share an equal rank, then Team C deserves to share the same rank because of its equality with B. Therefore, either A and C share the same rank, or A and B cannot share the same rank. Because the confidence bounds of A and C do not overlap, the former cannot be true, so A, B, and C will each be ranked separately (1, 2, and 3, respectively).

I think each can have arguments made for them, but I'm curious if there is a literature which has addressed the merits of these (and perhaps other) approaches to grouping rankings by using confidence intervals.

Answer 1

Not disagreeing with @Kontorus, but putting in some more context.

You are making "multiple comparisons"; A with B, A with C, B with C. In such circumstances overlapping groups are common, often indicated by grouping symbols e.g.

              value                      Grouping
1 (Team A)    0.1234 (CI 0.1232-0.1236)  1
2 (Team B)    0.1232 (CI 0.1228-0.1236)  1 2
3 (Team C)    0.1229 (CI 0.1227-0.1231)    2

Values that do not share a grouping number are significantly different

As @Kontorus remarked Minitab produces such "Grouping Information Tables" from multiple comparison tests following ANOVA, but you can use a similar display more generally.

There is a lot of information out there on multiple comparisons, you might want to read up on it. You should be aware that making multiple comparisons increases the risk of Type 1 errors.

Edited to add: you should also be aware that non-overlapping confidence intervals is not the same as a significance test of difference; see http://www.ncbi.nlm.nih.gov/pmc/articles/PMC99228/

Edited to add: this may be of interest "An Algorithm for a Letter-Based Representation of All-Pairwise Comparisons" http://www.tandfonline.com/doi/abs/10.1198/1061860043515

Answer 2

You could just rank them in the groups you've already created: A and B are both in rank 1, and B and C are both in rank 2.

The problems with your current rankings systems are:

1. Ranking them all the same ignores the fact that A and C are probably different.
2. Ranking A and B separate from C ignores the fact that B and C could be the same.
3. Ranking A, B, and C separately ignores the fact that B could in fact be the same as A or C. Placing B in it's own ranking gives the false impression that it might not be in the same ranking as A or C.

I know for a fact that groupings like these exist in Minitab, but I'm not aware of any literature that discusses this.