Test for multiple samples of ordinal data

Question

I have 10 independent samples of ordinal data ranked 1-3. I also have predicted ranks for the same data set. I want to test if the predicted ranking matches the observed ranking at a rate better than chance. A similar example might be a set of 10 races among 3 contestants where each race has different people in the races and we want to see if we can predict the order they finish in better than chance. Our data might look like so:

| Race Number | Pred Place 1st | Pred Place 2nd | Pred Place 3rd |
|-------------|----------------|----------------|----------------|
|      1      |        2       |        1       |        3       |
|      2      |        1       |        3       |        2       |
|      3      |        3       |        1       |        2       |
|      4      |        3       |        2       |        1       |
|      5      |        3       |        2       |        1       |
|      6      |        3       |        1       |        2       |
|      7      |        1       |        2       |        3       |
|      8      |        1       |        2       |        3       |
|      9      |        1       |        3       |        2       |
|     10      |        1       |        2       |        3       |

Where Race Number is an indicator for which race we are looking at Pred Place 1st is the predicted place for the contestant who actually came in first in that race, Pred Place 2nd for the one who came in second, etc.

I know for a single paired ordinal sample the Wilcoxon signed-rank test would be used, but I am not sure what to do with multiple samples. Would this be a good application of Fisher's method? Also with only 3 values to rank, my Wilcoxon test only returns p-values of 1 or NaN, is there another test that would be more appropriate here?

Answer 1

Scoring method. Your scoring method seems to be the number of places correctly picked. So 123 gets 3 points, 321 gets 1 point, 213 gets no points, etc. It's your project, and you're entitled to your own scoring method, but I mention another scoring method below for your consideration.

Note: You do not how exactly how you did your Wilcoxon test. The thought has crossed my mind that you're not doing it correctly. So let's take a further look at the scores and the one-sample Wilcoxon test.

This scoring system, as I understand it, can be programmed in R (where 1:3 is short for vector c(1,2,3) as follows:

sum(c(1,2,3)==1:3)
[1] 3
sum(c(1,3,2)==1:3)
[1] 1
sum(c(2,1,3)==1:3)
[1] 1
sum(c(3,1,2)==1:3)
[1] 0

To me, an anomaly seems to be that 321 (exactly the wrong order) gets a better score than 312 (which at least puts 1 in second place).

Distribution of scores for guessing. It is not difficult to see that random scoring would give $P(X = 0) = 1/2, P(X = 1) = 1/3, P(X = 3) = 1/6),$ so that $E(X) = 1.$

A simulation of a million random scores (with 2 or 3 place accuracy for evaluated probabilities) goes as follows:

set.seed(727);  m=10^6
x = replicate( m,  sum(sample(1:3,3)==1:3) )
table(x)/m
x
       0        1        3 
0.333372 0.500043 0.166585 

Example of Wilcoxon test. If your ten scores $X_i$ are given by the vector x = c(3,1,3,0,3, 3,3,1,1,0), then the Wilcoxon signed rank test, that the scores are significantly better than $1,$ gives P-value 0.032 (with cautionary notes about 0's and ties among the data) and a one-sided t test gives P-value 0.043. (With the small sample size and highly discrete data, one is entitled to question whether the t test is sufficiently robust to be useful.)

This vector of scores is hypothetical--not taken from the data in your Question.

x1 = c(3,1,3,0,3, 3,3,1,1,0)
wilcox.test(x1, mu = 1, alt="g")

        Wilcoxon signed rank test with continuity correction

data:  x1 
V = 25, p-value = 0.03249
alternative hypothesis: true location is greater than 1

Warning messages:
1: In wilcox.test.default(x1, mu = 1, alt = "g") :
  cannot compute exact p-value with ties
2: In wilcox.test.default(x1, mu = 1, alt = "g") :
  cannot compute exact p-value with zeroes

t.test(x1, mu=1, alt="g")

    One Sample t-test

data:  x1
t = 1.9215, df = 9, p-value = 0.04342
alternative hypothesis: true mean is greater than 1
95 percent confidence interval:
 1.036814      Inf
sample estimates:
mean of x 
      1.8 

Alternative scoring method. Personally, I would prefer giving credit for having adjacent orders correct, and being docked for incorrect adjacent orders, so 123 gets 2 points 132 gets 1 point, 312 get -1 point, and 321 gets -2 points, etc.

This system can be programmed in R as shown by examples below:

sum(diff(c(1,2,3)))   # differences are 1 & 1
[1] 2
sum(diff(c(1,3,2)))   # differences are 2 and -1
[1] 1
sum(diff(c(3,1,2)))   # differences are -2 and 1
[1] -1
sum(diff(c(3,2,1)))   # differences are -1 and -1
[1] -2

Distribution of scores for guessing. Using this system $P(Y = -2) = 1/6, P(Y=-1)=1/3,$ $P(Y = 1) = 1/3,$ $P(Y = 2) = 1/6.$ So that $E(Y) = 0.$

A simulation of a million random scores (with 2 or 3 place accuracy) goes as follows:

set.seed(727);  m=10^6
y = replicate( m,  sum(diff(sample(1:3,3))) )
table(y)/m

y
      -2       -1        1        2 
0.166547 0.333372 0.333496 0.166585 

If my ten $Y_i$ scores are `y = c(2,1,2,-1,2, 2,2,1,1,-1),$ then a Wilcoxon signed rank test gives P-value 0.014. (There are some ties, but no 0's.) A t test gives P-value 0.009. (The sample size is as above, but there are more possible values of scores here.) So both tests find that this sequence of scores is better than random guessing.

y = c(2,1,2,-1,2, 2,2,1,1,-1)
wilcox.test(y, alte="g")   # center 0 assumed

     Wilcoxon signed rank test with continuity correction

data:  y
V = 49, p-value = 0.01396
alternative hypothesis: true location is greater than 0

Warning message:
In wilcox.test.default(y, alte = "g") :
  cannot compute exact p-value with ties

t.test(y, alte="g")

        One Sample t-test

data:  y
t = 2.9055, df = 9, p-value = 0.008719
alternative hypothesis: true mean is greater than 0
95 percent confidence interval:
 0.4059946       Inf
sample estimates:
mean of x 
      1.1 

Of course, there are many additional possible scoring methods. The point is that you need to decide on a method that makes sense to you and then apply it correctly.