0
$\begingroup$

I've done a bit of reading and I'm more confused than I started. What is the correct way to build a classification (binary) model that doesn't give overly optimistic (or pessimistic) results.

Suppose I have a data set of 7000 samples with around 700-800 features. The classes are about 70/30 biased towards the positive class.

I've been using an SVM whose parameters I preset and doing 10-fold CV. I then take mean and variance of the false positive rate and the false negative rate as my model performance metric.

I now would like to do a grid search on the parameters (which will likely entail another inner cross validation). I don't think doing it on the entire set first is valid since it breaks the cross validation independence but if I do it for each of the folds I'll have 10 different models.

What is the correct workflow for training classifiers when the sample size isn't big enough to split into multiple pieces?

$\endgroup$
  • $\begingroup$ What do you mean by "The classes are about 70/30 biased towards the positive class." ? $\endgroup$ – utobi Oct 28 '16 at 13:17
  • 1
    $\begingroup$ 70% of my samples have label 1 and the other 30% are labeled -1 $\endgroup$ – user27108 Oct 28 '16 at 13:29
1
$\begingroup$

I don't think doing it on the entire set first is valid since it breaks the cross validation independence

That thought is correct: you want to look into nested cross validation
(In principle you can nest all kinds of validation schemes: nesting of single splits leads to the typical training + optimization/hyperparameter tuning (aka validation set) + validation of final model (aka test set) setup).

but if I do it for each of the folds I'll have 10 different models.

Yes and no. Yes, in the outer cross validation you generate a number of tuned surrogate models. But if all is well with your models, they should end up with the same hyperparameters, and there really shouldn't be any decisions or choice involved:

One of the key assumptions for cross validation that the modeling is stable, leading to equivalent (if not equal) surrogate models - which are in turn assumed to be equivalent to the "final" or "big" model trained (using the same tuning routine) on the whole data set: if you observe instability already among the surrogate models, extrapolation of performance characteristics to the final model is a shot into the dark.

So if you can show that the tuned surrogate models are stable including their hyperparameters (i.e. the tuned surrogate models have the same hyperparameters although the tuning was done separately for each surrogate model) you're fine.
A nice side effect of the cross validation is that you can check whether the tuning is in fact stable: if it isn't you have deeper problems and need to rethink your modeling approach (constraints/regularization/fixing hyperparameters externally).

* Things get more difficult if you can have different but equivalent sets of hyperparameters, i.e. the hyperparameter space has several equivalent minima.

$\endgroup$
0
$\begingroup$

My suggestion would be:
1. Use the median F1 score on each combination of parameters and CV-fold. See: https://en.wikipedia.org/wiki/F1_score
2. Of each of the 10 F1-scores you get for each parameter set, take the median (average of the middle 2 scores).
3. Pick the parameter set with the highest median F1-score.

$\endgroup$
0
$\begingroup$

I think the easiest answer to your question is in Chapter 9 of the book by James et al. (2013) Introduction to Statistical Learning with Applications in R. You can get the pdf of the book (for free!) form Gareth James's webpage http://www-bcf.usc.edu/~gareth/index.html. The authors also provide an R implementation.

However, in your case I would go with a random forest. It is much easier to tune (R can do it for you almost automatically!) and it performs well in practice. Hope it helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.