2
$\begingroup$

I’m having problems with understanding the relation between model evaluation, application of cross validation, and the use of a test set.

I don’t think that my used modelling approach (Bayesian networks) is very important for the content of the question, still I quickly explain it to give the question a practical meaning: I want to use Bayesian networks as a tool for classifying my data. I run three approaches: A with a manually built structure, B and C with algorithmically learned structures. So, for people that don’t know so much about Bayesian networks: these are different approaches how I structure dependencies between variables – but no variable selection is performed, all classifiers have the same amount of variables used.

My modelling process is as follows:

At first, I separate the test set (25%) to spare some unseen data (I have more than enough data points). The remaining 75% are used for 10-folds Cross Validation (CV). There are different explanations of the use of CV, some speak of “tuning of hyperparameters”, some about “model selection”, others just about “model evaluation”.

I want to use CV to get an estimate of my expected prediction error (see Elements of statistical learning, p.241), so for every fold I calculate Precision, Recall and other measures. Averaging these values for every model gives me a performance ranking. I don’t really want to perform neither feature nor model selection as I only want to compare these three approaches.

But that’s where I struggle – how do I continue with my test set? I could take the best classifier of the three, and let it predict the test set. This shows me how it performs on unseen data. But what do I do with the other two classifiers? As I only want to compare models, it wouldn’t it also be interesting to check their performance on unseen data? The gold standard of the test set - don't use the test set for model selection - wouldn't be violated, as I don't perform model selection.

Some practical things still are also unclear – what parameters for my best classifier should I use? The ones that best performed in the CV? I could also discard all models of the CV, relearn on the whole 75% of my data and then predict my test set – this would somehow make the CV pointless.

Maybe my approach doesn’t even need a test set?

I would be grateful for any help. There is a lot of literature on CV, also a lot of posts throughout this network and the internet, still I can’t find answers to my questions. Thank you!

$\endgroup$
  • $\begingroup$ On one hand, you talk about the best classifier and at the same time you say you don't do any model selection. From my point of view, there is a contradiction here. Can you please clarify what is the intended outcome of the whole exercise: do you want to be able to select one (best) model or you just want to get performance estimates for all three models? $\endgroup$ – aivanov Jul 6 '18 at 14:45
0
$\begingroup$

The goal of the cross-validation is to reduce overfitting. I'm pretty sure this is something you already know, but let's go in more details :

If you were to use 100% of your data to train your algorithm, obviously you would overfit, that's bad.

Now, we may wonder why is there a cross-validation and a test set? The thing is, you'll probably end up overfitting a little bit on your cross-validation. In comes the test set, this makes sure that you don't end up with a cross-validation set that would be better off in the training set.

You say you don't want to do model/feature selection because you only want to compare approaches. The thing is, one model may perform drastically better than the others with the right features. I understand your POV that, if I understood correctly, training all models with the same parameters would give you the best approach. But what if you don't have the right set of features for an optimal solution?

I have read on Kaggle that you should always base your approach on your CV results. The reason is simple, if you test set gives you a bad accuracy, you'll end up modifying your hyperparameters/model/etc to gain accuracy on test model and you'll end up overfitting your test set. The cross-validation set, being "always" different because of the k-fold, gives you a better idea of the overall performance of your algorithm.

The test set should just be there at the end to make sure you didn't mess up and act as a reflection of unknown data. If you keep on testing on it and adjusting your parameters to the test set result, it isn't unknown anymore. But you should forbid yourself to act too much on the test set. Keep it far.

I think you should test your three models on the test set at the end. But maybe you should also consider stacking your models, which may yield an even better performance.

To answer your last question, I believe it isn't a bad idea to train on your full data set once and only once you are done with your training. At this moment, the extra data might help you get a little bit more accuracy, but I don't believe there are papers that prove this point. It's just a known fact that more data = better results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.