I've done a bit of reading and I'm more confused than I started. What is the correct way to build a classification (binary) model that doesn't give overly optimistic (or pessimistic) results.
Suppose I have a data set of 7000 samples with around 700-800 features. The classes are about 70/30 biased towards the positive class.
I've been using an SVM whose parameters I preset and doing 10-fold CV. I then take mean and variance of the false positive rate and the false negative rate as my model performance metric.
I now would like to do a grid search on the parameters (which will likely entail another inner cross validation). I don't think doing it on the entire set first is valid since it breaks the cross validation independence but if I do it for each of the folds I'll have 10 different models.
What is the correct workflow for training classifiers when the sample size isn't big enough to split into multiple pieces?