3
$\begingroup$

Is it possible to determine whether one model's Spearman rank is significantly better than that of another model based on the p-values for each?

Let's say that I have

model 1: $\rho_s$ = 0.9, p-value = 1.5E-10

model 2: $\rho_s$ = 0.86, p-value = 1E-10

Can I reject or accept the hypothesis that model 1 has a better Spearman rank than model 2 based on a criteria/formula/method which involves their p-values?

Background

The models make predictions of a molecular property that I know from experiment. I have results for 50 molecules. So I have two sets of predictions of length 50, and one set of experimental results of length 50.

Edit

I have decided to simply Bootstrap the $\rho_s$. This gives a confidence interval I can live with.

$\endgroup$
6
  • $\begingroup$ I usually prefer to analyse the value of the correlation instead of the p-value, since the latter has a more visible dependence on the sample size. $\endgroup$ – Ertxiem - reinstate Monica Dec 20 '19 at 9:53
  • $\begingroup$ Right, but is it possible to say one correlation is better by using knowledge of the p-values? $\endgroup$ – Charlie Crown Dec 20 '19 at 9:57
  • $\begingroup$ I prefer to say that a correlation is stronger (better) because it has an absolute value closer to $1$. $\endgroup$ – Ertxiem - reinstate Monica Dec 20 '19 at 13:07
  • $\begingroup$ Numbers are meaningless without significance level though. How do i define statistically if one has a higher rank than the other, or at least prove that I can't reject that one is higher $\endgroup$ – Charlie Crown Dec 20 '19 at 17:53
  • $\begingroup$ Just in the same way p-values are meaningless without the corresponding statistical values. And p-values do not prove anything, they just correspond to a confidence level that the null hypothesis is true. Low p-values mean that it's unlikely that the null hypothesis is true, which is different from proving that it's false. $\endgroup$ – Ertxiem - reinstate Monica Dec 21 '19 at 2:28
0
$\begingroup$

No. For one thing, the sample sizes could be different. More goes into the p-value than the effect size.

$\endgroup$
2
  • $\begingroup$ The sample sizes are both the same. It is a set of measurements by two models compared to the same data set. $\endgroup$ – Charlie Crown Dec 20 '19 at 17:12
  • $\begingroup$ @CharlieCrown Note that when sample sizes are different, ranking of p-values is equivalent to rank correlations. $\endgroup$ – Ertxiem - reinstate Monica Dec 21 '19 at 2:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.