5
$\begingroup$

Is it possible to have test error lower than training error?

I have a classification problem with 2000 samples, 500 of which are positives, 1500 are negatives. I split my data into 70% training data, 30% test data.

Run random forest with 200 estimators and cv=10. I did this several times and compared the recall and precision score and notice the scores for my test set is significantly better. Is this possible?

$\endgroup$

migrated from stackoverflow.com Jul 10 '15 at 18:36

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ Try rotating the test data and training data. $\endgroup$ – shuttle87 Jul 10 '15 at 2:43
  • $\begingroup$ It can happen, especially since your data set is imbalanced. Try using k-fold cross validation or even stratified k-fold. $\endgroup$ – IVlad Jul 10 '15 at 8:17
  • $\begingroup$ @shuttle87. After rotating the test and training data, (ie. training on the test data, testing on the train data). The recall for the test data (the one I now trained on) remains inferior. $\endgroup$ – Jason Jul 13 '15 at 23:51
  • $\begingroup$ what data set are you using? $\endgroup$ – Charlie Parker May 16 '16 at 14:14
5
$\begingroup$

Totally possible, though it probably means that you aren't training quite as much as you could be. Typically when you look at test/train accuracies over time you get a graph like this:

Credits go to Daniel Nee

The test/train stages can be (very broadly) categorized as follows:

  • first you start training and the test/train accuracy is noisier, but they are very strongly correlated. This means you haven't quite fit to the problem.
  • As time goes on, they both start to decrease, but the training error starts to decrease more quickly than the testing error. This means you're approaching a very good level of fit.
  • Eventually you start to see the error rate of the testing set increase, while the training set error continues to decrease. This means you have officially started to overfit.

There are a lot of ways of dealing with overfitting if that becomes a problem, but your goal in picking an algorithm and training should be to hit the highest accuracy, which typically happens somewhere in the second stage.

If your test accuracy is higher than your train accuracy, you are likely still very far left on the training graph. There are three main options for resolving that problem:

  • use an algorithm better suited for small datasets (hard to tell without knowing about your problem, but Naive Bayes is usually a good small data choice)
  • Change your model constants to fit more strongly to your training set (increasing the learning rate)
  • Get more data
$\endgroup$
  • $\begingroup$ thanks for the answer @Slater Tyranus. I am doing a binary classification problem with 3000 rows, 40 variables. 1000 positives and 2000 negatives. Currently I am using RFC with n_est = 160, depth = None, features = 30. I've tried using AdaBoost and ExtraRandomTree both giving inferior results. $\endgroup$ – Jason Jul 13 '15 at 23:55
  • $\begingroup$ @Jason yea, that's a pretty tiny dataset. RFC might be decent depending on the problem, but with that amount of data I'd advocate trying Naive Bayes, or an SVM. If more data is an option then do that. Switching classifiers won't turn lead into gold. $\endgroup$ – Slater Victoroff Jul 14 '15 at 1:04
  • $\begingroup$ You said that it might mean that "one might need more training". However, say for methods that train in one single step like PCA or kernel methods, if one still observes higher training set than test set, how does your explanation above applies? $\endgroup$ – Charlie Parker May 16 '16 at 13:39
  • $\begingroup$ @CharlieParker if it trains in one step and you're still seeing this behavior it likely means you either need more data, or to change the approach you're taking. In general this is reflective of a model that's still learning, or a dataset with a low signal to noise ratio. Some of these may just not be tractable problems, but if you see this with a model that won't support further training you should probably switch models. $\endgroup$ – Slater Victoroff May 16 '16 at 17:38
  • $\begingroup$ @SlaterTyranus I am using the MNIST data set. I've been told that the data set is "too easy". So I tried the alternative MNIST with background noise and this issue stopped, now the test is higher. Though, I am not sure if I artificially fixed the problem or not. I thought that if the task is "too easy" then it means that even after training, since training set is larger than test set, it is possible the model extracted all info and when it saw a small data set, it was lucky enough to perform really well on it. But I guess its a good idea to also test a different model though. $\endgroup$ – Charlie Parker May 17 '16 at 5:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.