This might sound like a very simple question, but I haven't been able to find an answer to it, yet: Assuming I am working on a binary classification task and I am using k-fold cross-validation to train classifiers on a train/test split of my data: After running CV I would end up with k trained classifiers. Which one of these would I actually use to classify new data? Would I simply meassure F1 etc. and use the best one? Or should I use some kind of ensemble method and use all k in some kind of voting?
Short answer: Use the best model settings selected from cross validation to train a final model on your entire training set.
Cross validation is used for two purposes:
- Model selection
- Model evaluation
Model selection is when you are comparing competing models, this could be in the form of different architectures, i.e. SVM vs kNN vs random forest, or one algorithm with different hyper-parameters, i.e. SVM with linear kernel function, but 5 different values of cost. You can even use it to compare multiple architectures with multiple hyper-parameter values, i.e. linear SVM with 5 values of c vs kNN with 3 values of k.
Typically you will evaluate each competing model under cross validation, and then select the model setup with the highest cross validation score (the average of its scores on each of the k folds) to be your final model. Say I was comparing the 5 SVMs and the 3 kNN models mentioned in the last paragraph. I discover that an SVM with c=0.01 had the highest CV score overall, this setup will form my final model. However, as you say, I have k implementations of this model as it's been formed on k folds. To obtain a single final model I fit a model using these settings on all the training data.
Model evaluation comes after model selection, and is used to provide an unbiased estimate of how well your final model will perform on unseen data. You may think that the cross-validation score calculated in the model selection phase does this, but it will be biased as you have selected these settings owing to their strong CV score. You'll probably need to use nested cross-validation for this, see the excellent answer by cbeleites here for a better explanation than I can provide.
$k$-fold cross-validation is a method to estimate generalization performance, for a given modeling strategy. You don't typically select one of these $k$ classifiers as your final model. In fact, the empirical best model during your cross-validation procedure is not necessarily really the best (for instance maybe its test fold happened to be an easy one).
Typically a new model is trained under the same circumstances on the full training set, or the $k$ classifiers you obtained during cross-validation are aggregated (for instance through bagging).
You don't necessarily pick any of the k 'models'.
You evaluate one model on k folds to get its average performance across (train, test) splits.
So, in this classification case, if your evaluation metric is let's say auc, then you get the average auc across k folds. You can compare this with another k-fold run with some other hyper-parameter's configuration on the same model / different model. From here, you can choose the best hyper parameters / model.
So the basic need of k-fold CV is to evaluate your model's performance across train test sets to get the most confidence, otherwise, if you only evaluate it across one set, then you cannot say with much confidence how your model will perform across a different set of train test.
A proper approach is not to train k various classifiers (on k folds) but rather one classifier k times. One does it to measure its performance in a robust way, always using a different fold as a test dataset. The resulting k performance measures are then aggregated in some way to give that robust performance metric m.
The variation of the classifier hyperparameters happens outside of the CV step. A separate CV step (with all k folds) is then run for each of all permutations of the classifier hyperparameters. The combination of hyperparameters that result in the best metric m is thus chosen and the classifier with those parameters may now be (re) trained on the full dataset.
So the master plan for workflow does not include any voting or otherwise combining the classifiers.