I am using about 256 predictors and target is sales. I am using a software called Alteryx which is R based. I have tried to run Random Forest, Spline model and Neural nets on same data.

I used partitioning to create test and training data sets.

I am fairly new to field, hence I am not sure how to compare performance of these models.

I was advised to compare Pearson correlation coefficient between predicted and actual value for all models and select the best one.

Although I am new, from what I have learned, I don't think its a good way to compare models.Pearson for Forest model is about 0.98 and for neural nets it's about 0.91. both figures are for testing set.

Is there any better way to compare and cross validate models?

Should I use R squared value/ AIC to compare?Is there any other manual way to compare models?

I have predicted and actual values for training data sets for all models, is there any way to compare using those?

  • 1
    $\begingroup$ Is this cross-validated accuracy? I ask because your model will obviously perform very well when you test it on the same data that you trained it on... $\endgroup$ – Sycorax says Reinstate Monica Apr 7 '15 at 15:09
  • 2
    $\begingroup$ See my answer here: stats.stackexchange.com/a/112052/1569 $\endgroup$ – Hong Ooi Apr 7 '15 at 15:12
  • $\begingroup$ Please provide more info on your data and experimental setup. $\endgroup$ – Vladislavs Dovgalecs Apr 7 '15 at 15:58
  • 1
    $\begingroup$ possible duplicate of Random Forest - How to handle overfitting $\endgroup$ – Sycorax says Reinstate Monica Apr 7 '15 at 16:29
  • $\begingroup$ In such cases, when predictive accuracy is implausibly high, it's worth checking whether any predictors are actually part and parcel of the outcome. E.g., if Y = Sales and one predictor is Sales in Region A. $\endgroup$ – rolando2 Jul 13 '18 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.