I have a dataset with 70 features, which are continuous measures and are interrelated but not highly correlated ($|\rho| <.5$. I have several outcomes, which are each integer values ranging from 0-80. For each outcome I perform the forllowing steps:
- I perform a random 50/50 train/test split.
- I fit a RF model using the randomForest package in R using the default settings.
- I predict score using the test data and calculate Pearson correlation between the predicted values and the outcome in the test set.
- I repeated this 100 times.
What I get get is a distribution of Pearson $\rho$ values that indicate the accuracy of the model where highest correlation of 1 means that the model perfectly predicts and 0 means the models does not predict at all.
What I get for some outcomes is a distribution of Pearson $\rho$ values where the distribution is smaller than 0. Thus, the model predicts basically "the opposite". I cannot make sense of this and therefore wanted to ask here what that means or how this could be? How can the correlation be significantly negative? If the model does not perform well or there is nothing in the data it should be randomly distributed or distributed around 0 but not significantly negative.
EDIT: I tested tuning mtry. Even though this improves out of sample accuracy for all models that worked anyways, it does not change the problem that some models predict the opposite. I also tried extremely random forests but get the same problem.