If i train my model using 70% of the data, and then test on the remaining 30% ... I am getting 65% accuracy predicting on the test set.
Out of interest i predicted on the training set (the same data from which i trained the model). I get roughly the same accuracy score (65%) with Logistic Regression. With Random Forest i get 80% predicting on the training set and around 65% predicting on the test set. So, in summary, with Logistic Regression i get consistent number whether i predict on the training set or the test set. With Random Forest i get a higher number when i predict on the training set. The accuracy from both models predicting on the test set is the same.
I am trying to understand which model is doing better. Should predicting on your training set always throw out a very a high accuracy? Or would that indicate overfitting?
Can I learn something from these stated measures? Perhaps i can conclude that the Random Forest is overfitting because accuracy from predicting on the training set is so much higher accuracy than predicting on the test set (not withstanding predicting on the test set is giving a reasonable number)?
If the model generalises well should the accuracy of predicting on training set and test set, be roughly the same? Is that in itself a good test for "generalising well", assuming the accuracy is a decent number?
Any input gratefully received.