0
$\begingroup$

I have a dataset with 70 features, which are continuous measures and are interrelated but not highly correlated ($|\rho| <.5$. I have several outcomes, which are each integer values ranging from 0-80. For each outcome I perform the forllowing steps:

  1. I perform a random 50/50 train/test split.
  2. I fit a RF model using the randomForest package in R using the default settings.
  3. I predict score using the test data and calculate Pearson correlation between the predicted values and the outcome in the test set.
  4. I repeated this 100 times.

What I get get is a distribution of Pearson $\rho$ values that indicate the accuracy of the model where highest correlation of 1 means that the model perfectly predicts and 0 means the models does not predict at all.

What I get for some outcomes is a distribution of Pearson $\rho$ values where the distribution is smaller than 0. Thus, the model predicts basically "the opposite". I cannot make sense of this and therefore wanted to ask here what that means or how this could be? How can the correlation be significantly negative? If the model does not perform well or there is nothing in the data it should be randomly distributed or distributed around 0 but not significantly negative.

EDIT: I tested tuning mtry. Even though this improves out of sample accuracy for all models that worked anyways, it does not change the problem that some models predict the opposite. I also tried extremely random forests but get the same problem.

Improve this question
$\endgroup$
8
  • $\begingroup$ There's nothing magical about the default settings. Indeed, it could be that the trees are too rich and the ensemble is overfitting. What happens when you tune the model? $\endgroup$
    – Sycorax ♦
    Commented Apr 28, 2019 at 20:01
  • $\begingroup$ Did you randomly split the data into test and training sets? $\endgroup$
    – Monotros
    Commented Apr 28, 2019 at 21:13
  • $\begingroup$ @Ozan yes, I did random split using the sample function in R. $\endgroup$
    – Jaynes01
    Commented Apr 29, 2019 at 7:14
  • $\begingroup$ @Sycorax I did not try to tune the model because in my field of research the standard model is used and it can be difficult to get through if you deviate from that. I will check it out but I need some time for that. $\endgroup$
    – Jaynes01
    Commented Apr 29, 2019 at 7:14
  • $\begingroup$ What exactly can you not make sense of? Sounds like your models just don't work very well. $\endgroup$
    – Peter Flom
    Commented May 24, 2019 at 12:21

1 Answer 1

Reset to default
3
$\begingroup$

You write:

I repeated this 100 times.

and ask

How can the correlation be significantly negative? If the model does not perform well or there is nothing in the data it should be randomly distributed or distributed around 0 but not significantly negative.

If you have random noise (I am not saying you do have random noise, but just suppose) then you should get significant results 5% of the time - half of them negative and half positive. (Assuming you are using 5% as a cutoff).

Even if the true correlation is slightly positive, you can easily get significant negative results. It will happen less than 2.5% of the time, but it can still happen.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I see the idea behind your answer but I am not sure if I get it in this particular context. Lets say I generate a random variable x and y, calculate Pearson r. Then 5% of the times should be significant. But in my case: I compute a correlation between actual and predicted values. Thus, between $y_i$ and $\hat{y}_i$ whereby the model to predict used a random subset of the data. I did this btw now 1000x instead of 100 with similar results. n is 100-150 depending on the outcome. Would you still say that the entire cloud of 1000 correlations could be negative? $\endgroup$
    – Jaynes01
    Commented May 30, 2019 at 11:52
  • 2
    $\begingroup$ If there is no relation between active and predicted, then yes. So, either 1) You are messing up the code somewhere (I have no thoughts on that) or 2) Your predictions just aren't related much to the actual values. $\endgroup$
    – Peter Flom
    Commented May 30, 2019 at 12:37
  • 2
    $\begingroup$ It is actually likely that there is no relation between the features and the outcome. At least, any expected relation should be weak. I just falsely assumed that in such a case the distribution of Pearson values will be around 0. Thanks Peter $\endgroup$
    – Jaynes01
    Commented May 30, 2019 at 12:59
  • $\begingroup$ I btw simulated this 1e4 times: 1. simulate 120 random features and 1 random outcome, then for each of 1e4 datasets I repeated 1000 times the following: 2.1 take a random subsample of x and y, 2.2 train rf model, predict using test data and then compute correlation. 2.3 plot the cloud of 1000 correlations. So I end up with 1e4 plots that each show 1000 correlations. Not once was the entire cloud below zero or above zero. Rarely a big portion of the cloud is above/below zero but still a tail that covers the other side of zero. So, in addition to randomness, there were other factors at play. $\endgroup$
    – Jaynes01
    Commented Aug 24, 2020 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.