I’m having problems with understanding the relation between model evaluation, application of cross validation, and the use of a test set.
I don’t think that my used modelling approach (Bayesian networks) is very important for the content of the question, still I quickly explain it to give the question a practical meaning: I want to use Bayesian networks as a tool for classifying my data. I run three approaches: A with a manually built structure, B and C with algorithmically learned structures. So, for people that don’t know so much about Bayesian networks: these are different approaches how I structure dependencies between variables – but no variable selection is performed, all classifiers have the same amount of variables used.
My modelling process is as follows:
At first, I separate the test set (25%) to spare some unseen data (I have more than enough data points). The remaining 75% are used for 10-folds Cross Validation (CV). There are different explanations of the use of CV, some speak of “tuning of hyperparameters”, some about “model selection”, others just about “model evaluation”.
I want to use CV to get an estimate of my expected prediction error (see Elements of statistical learning, p.241), so for every fold I calculate Precision, Recall and other measures. Averaging these values for every model gives me a performance ranking. I don’t really want to perform neither feature nor model selection as I only want to compare these three approaches.
But that’s where I struggle – how do I continue with my test set? I could take the best classifier of the three, and let it predict the test set. This shows me how it performs on unseen data. But what do I do with the other two classifiers? As I only want to compare models, it wouldn’t it also be interesting to check their performance on unseen data? The gold standard of the test set - don't use the test set for model selection - wouldn't be violated, as I don't perform model selection.
Some practical things still are also unclear – what parameters for my best classifier should I use? The ones that best performed in the CV? I could also discard all models of the CV, relearn on the whole 75% of my data and then predict my test set – this would somehow make the CV pointless.
Maybe my approach doesn’t even need a test set?
I would be grateful for any help. There is a lot of literature on CV, also a lot of posts throughout this network and the internet, still I can’t find answers to my questions. Thank you!