0
$\begingroup$

I'm currently evaluating a machine-learned ranking task. Two (among other) performance measures are Spearman's Rho and Kendall's Tau (rank or concordance correlation). I'm basically comparing a ground-truth with my prediction results, represented by a 5-point vector, e.g. [1 2 3 4 5] and [1 3 2 4 5].

Now, my adviser told me to take the P-Values into account. As far as I understood (please correct me if I'm wrong), small P-values (<0.05) indicate that my result is significant.

Unfortunately I got high P-values, approximately 0.23 in average.

Now, my basic question is: What can I conclude? Are P-values even relevant to my task? Is concordance correlation maybe not the correct performance measure in this case?

Thanks for your help.

$\endgroup$
  • $\begingroup$ As "significant" you consider p < 0.05 (or < 0.01, or < 0.001). If you want to make decisions based on p-values you should stick to the logic. Bigger p-values tell you only that the results are not significant. $\endgroup$ – Tim Dec 17 '14 at 11:23
  • $\begingroup$ "random sampling" is a prerequisite for the test; it is not being tested itself. What is being tested is whether there is monotonic association. $\endgroup$ – Nick Cox Dec 17 '14 at 11:31
  • $\begingroup$ If your prediction results are just ranks, then identity of ranks and identity of values are one and the same, although ties as usual complicate discussion. However, it may be that you need to explain in much more detail how your "prediction results" are obtained. $\endgroup$ – Nick Cox Dec 17 '14 at 11:33
  • $\begingroup$ With Spearman rank one minimal change of ranks (e.g. 1 2 3 4 5 and 1 3 2 4 5) still yields a P-value of about 0.04 but two such changes yield a P-value of about 0.10. The first is conventionally significant; the second isn't. However, if you have good arguments for your own conventions, spell them out. The long and the short of it is that 5 data pairs are very little information on which to judge association and the difficulty of getting low P-values is entirely consistent with that; it's what P-values are supposed to do for you. $\endgroup$ – Nick Cox Dec 17 '14 at 11:38
  • $\begingroup$ That's already very helpful! Why is it important to know, how the prediction results are obtained? I'm using different machine-learning approaches, based on svm, mlr and listnet... it is kind of a quality ranking, better quality, higher rank... $\endgroup$ – Carsten Dec 17 '14 at 11:52
1
$\begingroup$

I assume you want to compare how good the machine's ordering meets the "real" ordering. In this case, a high rank correlation between truth and machine's output would indicate a successful machine. But how much is enough? The $p$-values to testing of no rank correlation are smaller the smaller the $p$-value is. 0.23 tells you that if the machine's ranking would be completely random (what you know is not true!), the probability of getting results as and more closely (in a probabilistics sense) to the true ordering is 0.23.

As the size of your experiment is each time only $n=5$, you will only get $p<0.05$ iff your machine gets the exact ordering. Even if you swap only two adjacent values, you would only get $p=0.083$. Sadly, only $p<0.05$ would usually be considered as evidence against random ordering of your machine, although this threashold can be questioned. So yes, your approach is not appropriate.

If you want a statistical test, you should rather take the orderings as random samples from the symmetric group $S_5$. Then you can realize at least some sufficient sample size by running your algorithm several times. Unfortunately, I don't know of statistical tests for samples from the symmetric group.

So for the time being, I would use an appropriate mapping of your orderings to a scalar. A metric on $S_5$ or the $p$-values could do the job. Then I would run the algorithm several times with random input and map the found orderings to this scalar. Then I would test if the mean of these sampled scalars is within some appropriate interval around the map of [1 2 3 4 5]. E.g. if you are willing to tolerate adjacent swappings and use the $p$-value as mapping, a mean $p$-value of 0.083 would belong to this interval.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.