- Model selection and cross-validation: The right way
- Crossvalidation and/or testdata. Always use both or can one exclude the other?
but I still don't get it.
My problem is to construct a simple SVM, tune it's parameters and calculate the generalization error. Assume I have a dataset with e.g. 10 features and 200 samples. I don't want to waste data, since it's relatively small. My approach would be:
Split the dataset (e.g., 70/30) with holdout method into training / validation and test set.
Make a repeated n-fold cross-validation on the training set. I calculate the error rate (misclassification rate) after the folds are done (simply counting all misclassified samples). I try to minimize the error rate (or any other loss function?) each time the complete n fold is done. I store the error rate and choose the parameters with the lowest error rate. Then I retrain the model on the complete training set with the estimated parameters.
- Calculate generalization error with the test set.
- The larger my test set is, the smaller gets the train set, so I discard potential information. Can this be solved via a "stacked" n-fold cv?
- Do I really have to make a REPEATED n-fold cv? Are there other possibilities?
- Is the error rate an appreciate loss function or should I choose another one (eg. the empirical error function or mse, but then I'd need a probability output, right?)?