3
$\begingroup$

If i train my model using 70% of the data, and then test on the remaining 30% ... I am getting 65% accuracy predicting on the test set.

Out of interest i predicted on the training set (the same data from which i trained the model). I get roughly the same accuracy score (65%) with Logistic Regression. With Random Forest i get 80% predicting on the training set and around 65% predicting on the test set. So, in summary, with Logistic Regression i get consistent number whether i predict on the training set or the test set. With Random Forest i get a higher number when i predict on the training set. The accuracy from both models predicting on the test set is the same.

I am trying to understand which model is doing better. Should predicting on your training set always throw out a very a high accuracy? Or would that indicate overfitting?

Can I learn something from these stated measures? Perhaps i can conclude that the Random Forest is overfitting because accuracy from predicting on the training set is so much higher accuracy than predicting on the test set (not withstanding predicting on the test set is giving a reasonable number)?

If the model generalises well should the accuracy of predicting on training set and test set, be roughly the same? Is that in itself a good test for "generalising well", assuming the accuracy is a decent number?

Any input gratefully received.

$\endgroup$
0
6
$\begingroup$

Should predicting on your training set always throw out a very a high accuracy? Or would that indicate overfitting?

No, not always. The reason some models perform better than others is due to the bias/variance tradeoff. Logistic regression is a linear model, and hence is high bias. A random forest is capable of learning non-linear effects (low boas) of variables, but this comes at the cost of more variance. This increase in accuracy is likely due to the lower bias of random forests.

Can I learn something from these stated measures?

Not particularly, the training error is a poor measure of performance since you're testing on things the algorithm has already seen. The model has to fit the data as best as possible, so the training error is ultimately uninformative as to future performance.

If the model generalises well should the accuracy of predicting on training set and test set, be roughly the same?

Maybe. Models can generalize well and still be slightly overfit.

$\endgroup$
4
$\begingroup$

People measure predictive accuracy on the test set and not on the training set for good reasons. If both models deliver 65% on the test set, I'd say they're both of pretty much the same prediction quality, and the fact that the Random Forest (RF) has a much higher value on the training set wouldn't bother me much. It does say that the RF overfits the training set, but that in itself isn't a big problem if the test set performance is fine. Obviously it all to some extent depends on other things such as the size of your test set (if that's too small whatever you observe doesn't mean too much). Also the test set needs to be independent of the training set.

$\endgroup$
6
  • $\begingroup$ I have about 2000 rows in my training set using LR and RF .. do you think its big enough? I tried 10000 rows and it did not impact the accuracy of the model more than 0.5-1.0% , so i've kept it at 2000 rows so i can train the model frequently. $\endgroup$
    – brownie74
    Sep 9 at 8:21
  • $\begingroup$ @brownie74 My comment was on test set size. I can't comment on the necessary training set size as this depends on the model you're fitting. $\endgroup$ Sep 9 at 8:57
  • $\begingroup$ @Chistian I am using 70/30 split...test=30 $\endgroup$
    – brownie74
    Sep 9 at 21:56
  • $\begingroup$ Are you asking whether your test set is big enough? I'm confused, your first comment seems to suggest you can make "number of rows" larger if you want, so why wouldn't you? The number you have may be good enough to tell apart accuracies that are clearly different; if you want to differentiate between say 64.9% and 65.4%, more will be better. $\endgroup$ Sep 10 at 12:53
  • $\begingroup$ Generally, is 2000 rows enough for LR and RF (assuming 70/30 split)? It is not enough for a neural network, i have tried that (i have not tried using a GAN to interpolate more training data). More data is not necessarily better in my case, because my data is financial and when you go back too far, the very earliest data is fundamentally different to the most recent. $\endgroup$
    – brownie74
    Sep 11 at 13:03
0
$\begingroup$

Yes, the random forest is probably overfitting. Assuming that your training and test sets are drawn from the same distribution (shuffling the data before splitting is a good way to ensure this), the improved accuracy on the training set compared to the test set is due to characteristics of the training set rather than the data in general - the definition of overfitting.

In this case you may in theory be able to get better test set performance by preventing the overfitting, but it's hard to say for sure.

$\endgroup$
6
  • 1
    $\begingroup$ Larger training error has, in my opinion, been confused with overfitting. I don't think that is necessarily true. Overfitting would be when the model sees an increase in validation error due to too much complexity. OP may find themselves here, but OP would need to cross validate their random forest to detect this. $\endgroup$ Sep 8 at 13:30
  • $\begingroup$ @DemetriPananos The settings I am using for RF have been arrived at via a gridsearch (CV=5), so as far as I know, they are optimal (within the search space). Does this mean my RF has been cross validated? Is that what you mean? $\endgroup$
    – brownie74
    Sep 8 at 13:35
  • $\begingroup$ @brownie74 What I mean is that you need to cross validate over model configurations (i.e. hyperparameters). Random forests allowing for more splits are more complex, and if the random forest really is overfitting then the validation error should increase as more splits are made. $\endgroup$ Sep 8 at 13:40
  • $\begingroup$ @DemetriPananos The validation error is "1-accuracy" is it, when i predict on the test set? $\endgroup$
    – brownie74
    Sep 8 at 13:43
  • $\begingroup$ @brownie74 Yes, correct. $\endgroup$ Sep 8 at 13:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.