1
$\begingroup$

I have 10 independent samples of ordinal data ranked 1-3. I also have predicted ranks for the same data set. I want to test if the predicted ranking matches the observed ranking at a rate better than chance. A similar example might be a set of 10 races among 3 contestants where each race has different people in the races and we want to see if we can predict the order they finish in better than chance. Our data might look like so:

| Race Number | Pred Place 1st | Pred Place 2nd | Pred Place 3rd |
|-------------|----------------|----------------|----------------|
|      1      |        2       |        1       |        3       |
|      2      |        1       |        3       |        2       |
|      3      |        3       |        1       |        2       |
|      4      |        3       |        2       |        1       |
|      5      |        3       |        2       |        1       |
|      6      |        3       |        1       |        2       |
|      7      |        1       |        2       |        3       |
|      8      |        1       |        2       |        3       |
|      9      |        1       |        3       |        2       |
|     10      |        1       |        2       |        3       |

Where Race Number is an indicator for which race we are looking at Pred Place 1st is the predicted place for the contestant who actually came in first in that race, Pred Place 2nd for the one who came in second, etc.

I know for a single paired ordinal sample the Wilcoxon signed-rank test would be used, but I am not sure what to do with multiple samples. Would this be a good application of Fisher's method? Also with only 3 values to rank, my Wilcoxon test only returns p-values of 1 or NaN, is there another test that would be more appropriate here?

$\endgroup$
  • 1
    $\begingroup$ Why can't you combine all the data points into one sample? I think you should post what your data looks like. That may be helpful. $\endgroup$ – M Waz Jul 18 at 21:51
  • $\begingroup$ If you combine all your samples into one with an added factor variables showing which samples it is from, you can use a standard test for this. $\endgroup$ – user2974951 Jul 22 at 6:02
  • $\begingroup$ @user2974951 What standard test would you recommend? Wilcoxon doesn't have additional variables to the best of my knowledge. $\endgroup$ – Barker Jul 22 at 21:56
  • $\begingroup$ I am confused, what are these variables in your data Pred Place 1st to 3rd, what do they represent? Also, what is the observed rate? Do you have data for it? $\endgroup$ – user2974951 Jul 24 at 12:50
  • 1
    $\begingroup$ @user2974951 10 races are run. Each race has 3 contestants. Before the race you predict what place each contestant will take. At the end of the race you compare the predictions to the actual results. Pred Place 1st is the place you predicted for the contestant who actually came in first. It the value in that column is 1 you predicted correctly, otherwise you were wrong. Pred Place 2nd is the same for second place, all 2 predictions are correct. $\endgroup$ – Barker Jul 25 at 18:25
1
$\begingroup$

Scoring method. Your scoring method seems to be the number of places correctly picked. So 123 gets 3 points, 321 gets 1 point, 213 gets no points, etc. It's your project, and you're entitled to your own scoring method, but I mention another scoring method below for your consideration.

Note: You do not how exactly how you did your Wilcoxon test. The thought has crossed my mind that you're not doing it correctly. So let's take a further look at the scores and the one-sample Wilcoxon test.

This scoring system, as I understand it, can be programmed in R (where 1:3 is short for vector c(1,2,3) as follows:

sum(c(1,2,3)==1:3)
[1] 3
sum(c(1,3,2)==1:3)
[1] 1
sum(c(2,1,3)==1:3)
[1] 1
sum(c(3,1,2)==1:3)
[1] 0

To me, an anomaly seems to be that 321 (exactly the wrong order) gets a better score than 312 (which at least puts 1 in second place).

Distribution of scores for guessing. It is not difficult to see that random scoring would give $P(X = 0) = 1/2, P(X = 1) = 1/3, P(X = 3) = 1/6),$ so that $E(X) = 1.$

A simulation of a million random scores (with 2 or 3 place accuracy for evaluated probabilities) goes as follows:

set.seed(727);  m=10^6
x = replicate( m,  sum(sample(1:3,3)==1:3) )
table(x)/m
x
       0        1        3 
0.333372 0.500043 0.166585 

Example of Wilcoxon test. If your ten scores $X_i$ are given by the vector x = c(3,1,3,0,3, 3,3,1,1,0), then the Wilcoxon signed rank test, that the scores are significantly better than $1,$ gives P-value 0.032 (with cautionary notes about 0's and ties among the data) and a one-sided t test gives P-value 0.043. (With the small sample size and highly discrete data, one is entitled to question whether the t test is sufficiently robust to be useful.)

This vector of scores is hypothetical--not taken from the data in your Question.

x1 = c(3,1,3,0,3, 3,3,1,1,0)
wilcox.test(x1, mu = 1, alt="g")

        Wilcoxon signed rank test with continuity correction

data:  x1 
V = 25, p-value = 0.03249
alternative hypothesis: true location is greater than 1

Warning messages:
1: In wilcox.test.default(x1, mu = 1, alt = "g") :
  cannot compute exact p-value with ties
2: In wilcox.test.default(x1, mu = 1, alt = "g") :
  cannot compute exact p-value with zeroes

t.test(x1, mu=1, alt="g")

    One Sample t-test

data:  x1
t = 1.9215, df = 9, p-value = 0.04342
alternative hypothesis: true mean is greater than 1
95 percent confidence interval:
 1.036814      Inf
sample estimates:
mean of x 
      1.8 

Alternative scoring method. Personally, I would prefer giving credit for having adjacent orders correct, and being docked for incorrect adjacent orders, so 123 gets 2 points 132 gets 1 point, 312 get -1 point, and 321 gets -2 points, etc.

This system can be programmed in R as shown by examples below:

sum(diff(c(1,2,3)))   # differences are 1 & 1
[1] 2
sum(diff(c(1,3,2)))   # differences are 2 and -1
[1] 1
sum(diff(c(3,1,2)))   # differences are -2 and 1
[1] -1
sum(diff(c(3,2,1)))   # differences are -1 and -1
[1] -2

Distribution of scores for guessing. Using this system $P(Y = -2) = 1/6, P(Y=-1)=1/3,$ $P(Y = 1) = 1/3,$ $P(Y = 2) = 1/6.$ So that $E(Y) = 0.$

A simulation of a million random scores (with 2 or 3 place accuracy) goes as follows:

set.seed(727);  m=10^6
y = replicate( m,  sum(diff(sample(1:3,3))) )
table(y)/m

y
      -2       -1        1        2 
0.166547 0.333372 0.333496 0.166585 

If my ten $Y_i$ scores are `y = c(2,1,2,-1,2, 2,2,1,1,-1),$ then a Wilcoxon signed rank test gives P-value 0.014. (There are some ties, but no 0's.) A t test gives P-value 0.009. (The sample size is as above, but there are more possible values of scores here.) So both tests find that this sequence of scores is better than random guessing.

y = c(2,1,2,-1,2, 2,2,1,1,-1)
wilcox.test(y, alte="g")   # center 0 assumed

     Wilcoxon signed rank test with continuity correction

data:  y
V = 49, p-value = 0.01396
alternative hypothesis: true location is greater than 0

Warning message:
In wilcox.test.default(y, alte = "g") :
  cannot compute exact p-value with ties

t.test(y, alte="g")

        One Sample t-test

data:  y
t = 2.9055, df = 9, p-value = 0.008719
alternative hypothesis: true mean is greater than 0
95 percent confidence interval:
 0.4059946       Inf
sample estimates:
mean of x 
      1.1 

Of course, there are many additional possible scoring methods. The point is that you need to decide on a method that makes sense to you and then apply it correctly.

$\endgroup$
  • $\begingroup$ Thanks for the feedback. I actually am interested in ordinal scoring more along the lines of a c-index were any pair of points that is in the correct order wether they are adjacent or not, so for example, 213 would get "credit" for 2<3 and 1<3 and "docked" for 2<1. Can you explain why you think adjacent values are a better metric than the more standard pairwise comparisons? It doesn't seem beneficial to give more "points" for having two values that should be far apart close together so long as they are in the correct order like in a bigger dataset if 1 and 9 were adjacent. $\endgroup$ – Barker Jul 29 at 18:47
  • $\begingroup$ I was not trying to persuade you to use any one scoring method. Just trying to make the point that you need to understand what the distribution of the values (from whatever scoring system) under random guessing is. That way you can know what mean to specify in a Wilcoxon or t test and you can know whether you have enough different values to avoid massive ties. $\endgroup$ – BruceET Jul 29 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.