0
$\begingroup$

I separated my data set into three parts includes training, validate, and testing. I performed k-fold validation with using the validate set, then test the true performance of the predictive model using the test set.

However, I could see that the classification accuracy using the test(unseen data) is pretty low. (~15% lower than the validation accuracy.) Is this normal? Thanks.

$\endgroup$
0
$\begingroup$

It is natural. Cross-validation almost always lead to lower estimated errors - it uses some data that are different from test set so it will cause overfitting for sure. But the percentage of decrease is quite big, and if you have big sample size and it can not be explained with stochastic effects, I would suggest that your classification method is overcomplicated.

$\endgroup$
  • $\begingroup$ Thank for your answer, I have got a multiclass classification problem (3 classes) where there is no many significant features related to the target. I ended up having around 55% accuracy rate for my best model. My test accuracy reveals ~38%. I split 60/20/20. $\endgroup$ – Rapry Jan 24 '16 at 23:34
  • $\begingroup$ @Rapry But how many samples do you have? $\endgroup$ – German Demidov Jan 24 '16 at 23:35
  • $\begingroup$ I have 4,000 samples which is large enough (the optimal one). $\endgroup$ – Rapry Jan 24 '16 at 23:37
  • $\begingroup$ by the way, and selected model is not overtrainning. $\endgroup$ – Rapry Jan 24 '16 at 23:38
  • $\begingroup$ @Rapry try to choose not the best model (according to cross-validation), but the model with sub-optimal CV performance and less number of estimated parameters. Is your split completely random? How did you understand that selected model was not overtrained? I guess such decrease of performance directly indicate overtraining by definition. $\endgroup$ – German Demidov Jan 24 '16 at 23:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.