I have a dataset of a list of genes with predicted scores (of likelihood to cause disease) from 2 different machine learning classifiers:

Gene            Score1      Score2
RP11-983P164    0.2678077   0.2119513
SLC25A20        0.2644568   0.2586816
GLS             0.2560175   0.2631010
IKZF4           0.2468294   0.2189585
NRIP3           0.2446390   0.2170968
SENP1           0.2372014   0.2724868
SLC27A6         0.2321821   0.2218227
SRFBP1          0.2293986   0.2688244
OBFC1           0.2279012   0.2187441
STEAP2          0.2239941   0.2001475

I want to measure if any of the two predicted scores per gene are significantly different from each other, or if the predictions are very similar. I have a biology background so I'm not sure what to start with searching for this, and so sorry if I've asked this question in the wrong place, any help would be appreciated.


I now have 6 score columns in total (all look similar to Score1 and Score2) - are there any other statistical tests I can do? Would it be worth doing a t-test?

  • $\begingroup$ Do you have one observation per gene or do you have multiple? Maybe I'm wrong, but it seems hard to say something useful about the difference based on one observation. $\endgroup$ – J.C.Wahl Sep 28 '20 at 19:40
  • $\begingroup$ Are you wanting confidence or prediction intervals? Using that interval per prediction, you could find those predictions in which there isn't overlap and do additional investigation. $\endgroup$ – Chris Sep 29 '20 at 11:58
  • $\begingroup$ Another option is ANOVA of the 2 sets of predictions. But it really depends on what you are really trying to do. (Pairwise or as 2 sets) $\endgroup$ – Chris Sep 29 '20 at 12:09
  • $\begingroup$ Thank you for your comments. Honestly I am looking for any kind of measure that gives more detail to describe how these 2 predictions are different from each other. I actually will in a week or so have 3 more predictions columns (so 5 prediction scores per gene in total). I will look into ANOVA and interval per prediction - thank you for these, I don't have much of a stats background so have been trying mann–Whitney U from what I read but didn't have a lot of basis for doing it. $\endgroup$ – DN1 Sep 30 '20 at 8:09
  • $\begingroup$ So each predicted score column describes the likelihood to cause the same disease, or a different disease for each score column? Is each score column generated by a $different$ machine learning algorithm? If so, what are the algorithms? $\endgroup$ – develarist Oct 8 '20 at 15:58

One way to measure similarity is to estimate the correlation between Score1 and Score2. This will give you a number between -1 and 1 and the closer to 1 the higher the linear association between the scores.

If it is negative, then an increase in Score1 will give a decrease in Score2. This would mean that your models do not agree at all.

If the correlation is close to 0 then there is no linear relationship between Score1 and Score2.

  • 4
    $\begingroup$ Correlation doesn't measure similarity. It measures linear association. The columns could be on totally different scales and be highly correlated. Does that make them "similar"? $\endgroup$ – Do not reinstate Monica Sep 28 '20 at 13:13
  • 2
    $\begingroup$ Yes, they can be on different scales, but you can standardize them. From the data presented in the question, it does not look like that is a problem. What would you recommend as a starting point for measuring the similarity? $\endgroup$ – J.C.Wahl Sep 28 '20 at 14:02
  • $\begingroup$ I'm not sure, which is why I didn't answer. The definition of "similarity" here isn't very clear. Sounds like, for the OP, similarity means not being very different from each other, in an absolute sense. So, I guess something like mean squared error. $\endgroup$ – Do not reinstate Monica Sep 28 '20 at 14:08
  • 1
    $\begingroup$ @Do not, correlation (as well as its abs. value) can be seen as one of similarity measures. $\endgroup$ – ttnphns Sep 28 '20 at 14:09
  • 1
    $\begingroup$ @ttnphns, I have never heard that characterization but, regardless, it does not seem to comport with the kind of "similarity" the OP is talking about ("I want to measure if any of the two predicted scores per gene are significantly different from each other, or if the predictions are very similar.") $\endgroup$ – Do not reinstate Monica Sep 28 '20 at 14:24

It sounds like you just want a correlation matrix.

For x columns, this measures the correlation between each column's data.

Here, (Pearson's) correlation is a normalised version of the covariance of any two variables, so you don't need to worry about units.

In R, just read in your data frame with the 6 score columns. Since you want to check for significant differences, you can also do that with the Hmisc package, which gives significance levels (and yes, it uses the t-test for continuous scores).

# Just get correlation scores
cor_matrix <- cor(df)

# Get correlations *and* p-values of correlations for each pair
# Install Hmisc package first
cor_matrix2 <- rcorr(as.matrix(df))
cor_matrix2 # Gives a correlation matrix and a p-value matrix

Each element $x_{s1,s2}$ in the correlation matrix is $\in [-1,1]$, where 1 is perfectly correlated and -1 is perfectly inverse-correlated. Hence the diagonals will all be 1.

There are a number of assumptions made in calculating Pearson's correlation coefficient that you may or may not care about. E.g. if any of the data is ordinal, use Spearman's correlation coefficient instead; cor_matrix <- cor(df, method="spearman"). Check out the cor and rcorr help for more info on the R function and assumptions in general.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.