I am using simulation to compute a unique score for every college basketball team, and ranking these teams based on that score.
I am sensitive to the fact that the score sometimes differs by a tiny amount (1 part in 1,000), which is unlikely to be meaningfully different. Therefore, I am planning to use permutation to generate confidence intervals for these scores.
I planned to use the intersection of confidence intervals to determine if two teams had truly different rankings. For example:
1 (Team A) 0.1234 (CI 0.1232-0.1236) 2 (Team B) 0.1232 (CI 0.1228-0.1236) 3 (Team C) 0.1229 (CI 0.1227-0.1231)
In this example, however, I am uncertain about whether 2 teams should share the same rank, whether all 3 should, or whether they should all be ranked differently. Arguments for each:
Why to rank all the same:
Team A's confidence intervals overlap with those of Team B, and Team B's confidence intervals overlap with Team C. I think this can be dismissed because Team A's confidence intervals don't overlap with Team C, so it is clear that A and C should have different ranks, regardless of B's rank.
Why to rank A and B #1, and C #3:
Team A and B's confidence intervals overlap. Team A's and Team C's do not. We will define our ranking procedure to state that each rank terminates when the lower confidence bound of the team with the highest mean rank no longer overlaps with that of a subsequent team. In this case, this would cause A and B to tie for a rank of #1, and C to fall out into the next rank (#3).
Why to rank all separately:
Team A and B overlap, and team B and C overlap. If Team A and Team B share an equal rank, then Team C deserves to share the same rank because of its equality with B. Therefore, either A and C share the same rank, or A and B cannot share the same rank. Because the confidence bounds of A and C do not overlap, the former cannot be true, so A, B, and C will each be ranked separately (1, 2, and 3, respectively).
I think each can have arguments made for them, but I'm curious if there is a literature which has addressed the merits of these (and perhaps other) approaches to grouping rankings by using confidence intervals.