0
$\begingroup$

I have a dataset with 70 features, which are continuous measures and are interrelated but not highly correlated ($|\rho| <.5$. I have several outcomes, which are each integer values ranging from 0-80. For each outcome I perform the forllowing steps:

  1. I perform a random 50/50 train/test split.
  2. I fit a RF model using the randomForest package in R using the default settings.
  3. I predict score using the test data and calculate Pearson correlation between the predicted values and the outcome in the test set.
  4. I repeated this 100 times.

What I get get is a distribution of Pearson $\rho$ values that indicate the accuracy of the model where highest correlation of 1 means that the model perfectly predicts and 0 means the models does not predict at all.

What I get for some outcomes is a distribution of Pearson $\rho$ values where the distribution is smaller than 0. Thus, the model predicts basically "the opposite". I cannot make sense of this and therefore wanted to ask here what that means or how this could be? How can the correlation be significantly negative? If the model does not perform well or there is nothing in the data it should be randomly distributed or distributed around 0 but not significantly negative.

EDIT: I tested tuning mtry. Even though this improves out of sample accuracy for all models that worked anyways, it does not change the problem that some models predict the opposite. I also tried extremely random forests but get the same problem.

$\endgroup$
  • $\begingroup$ There's nothing magical about the default settings. Indeed, it could be that the trees are too rich and the ensemble is overfitting. What happens when you tune the model? $\endgroup$ – Sycorax says Reinstate Monica Apr 28 '19 at 20:01
  • $\begingroup$ Did you randomly split the data into test and training sets? $\endgroup$ – Gue Apr 28 '19 at 21:13
  • $\begingroup$ @Ozan yes, I did random split using the sample function in R. $\endgroup$ – Jaynes01 Apr 29 '19 at 7:14
  • $\begingroup$ @Sycorax I did not try to tune the model because in my field of research the standard model is used and it can be difficult to get through if you deviate from that. I will check it out but I need some time for that. $\endgroup$ – Jaynes01 Apr 29 '19 at 7:14
  • $\begingroup$ What exactly can you not make sense of? Sounds like your models just don't work very well. $\endgroup$ – Peter Flom - Reinstate Monica May 24 '19 at 12:21
2
$\begingroup$

You write:

I repeated this 100 times.

and ask

How can the correlation be significantly negative? If the model does not perform well or there is nothing in the data it should be randomly distributed or distributed around 0 but not significantly negative.

If you have random noise (I am not saying you do have random noise, but just suppose) then you should get significant results 5% of the time - half of them negative and half positive. (Assuming you are using 5% as a cutoff).

Even if the true correlation is slightly positive, you can easily get significant negative results. It will happen less than 2.5% of the time, but it can still happen.

$\endgroup$
  • $\begingroup$ I see the idea behind your answer but I am not sure if I get it in this particular context. Lets say I generate a random variable x and y, calculate Pearson r. Then 5% of the times should be significant. But in my case: I compute a correlation between actual and predicted values. Thus, between $y_i$ and $\hat{y}_i$ whereby the model to predict used a random subset of the data. I did this btw now 1000x instead of 100 with similar results. n is 100-150 depending on the outcome. Would you still say that the entire cloud of 1000 correlations could be negative? $\endgroup$ – Jaynes01 May 30 '19 at 11:52
  • 2
    $\begingroup$ If there is no relation between active and predicted, then yes. So, either 1) You are messing up the code somewhere (I have no thoughts on that) or 2) Your predictions just aren't related much to the actual values. $\endgroup$ – Peter Flom - Reinstate Monica May 30 '19 at 12:37
  • 2
    $\begingroup$ It is actually likely that there is no relation between the features and the outcome. At least, any expected relation should be weak. I just falsely assumed that in such a case the distribution of Pearson values will be around 0. Thanks Peter $\endgroup$ – Jaynes01 May 30 '19 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.