After doing the grid search for each surrogate model, you can and should check a few things:
- variation of the optimized parameters (here $\gamma$ and $C$).
Are the optimal parameters stable? If not, you're very likely in trouble.
- Compare the reported performance of the inner and outer cross validation.
If the inner (i.e. tuning) cross validation looks much better than the outer (validation of the final model), then you're in trouble, too: you are overfitting. There is a substantial risk, that the tuned parameters are not optimal at all. However, if the outer cross validation is done properly (all test sets are truly independent of the respective surrogate models), then at least you still have an unbiased (!) estimate of the model's performance. But you cannot be sure that it is optimal.
- How pronounced is the optimum? Does the performance degrade quickly for suboptimal parameters? How good is the optimal performance?
There's a whole lot to say about overfitting by model selection. However, it is good to keep in mind that both variance and optimistic bias can really hurt
- variance means that you may accidentally end up quite far away from the truly optimal hyper-parameters.
- but also bias can hurt: if you are overfitting, you can run into situations where many models look perfect to the inner cross validation (but they aren't really). In that case, the tuning can go astray because it doesn't recognize the differences between the models.
- If the bias depends on the hyper-parameters, you are in big trouble.
If you are interested in an example and you can read German, I could put my Diplom thesis online.
In my experience, tuning hyperparameters is an extremely effective idea for overfitting...
Now, if you realize that you are overfitting, you have mainly two options:
- report that the optimization had a problem with overfitting but
that you did a proper outer validation which resulted in ... (outer cross validation results).
- restrict the complexity of the model. One way of doing this is fixing hyper-parameters:
As an alternative to tuning the hyper-parameters to each training set, you could pre-specify (hyper)parameters (i.e. fix them beforehand). I do that as much as possible for my models as I usually have even fewer cases than you have, see below.
However, this fixing must really and honestly be done beforehand: e.g. I asked a colleague for his optimized parameters on a similar data set (independent experiment) or do a pre-experiment, including grid search on the parameters. That first experiment is then used to fix some experimental parameters as well as model parameters for the real experiment and data analysis. See below for further explanations.
Of course it is possible to do proper testing on automatically optimized models (double or nested validation), but your sample size may not allow splitting the data twice.
In that situation, it is IMHO much better to report an honest estimate for a model that was built using professional experience on how to choose modeling parameters than reporting an overoptimistic estimate on some kind of automatically optimized model.
Another point of view on the situation is that you have to trade off
- worse performance due to setting aside yet another bunch of cases for parameter optimization (smaller training sample size => worse model, but "optimal" parameters)
- worse performance due to suboptimal parameter fixing by the expert (but on larger training data).
Some similar thoughts on a similar question: http://stats.stackexchange.com/a/27761/4598
On the fixing of parameters and Dikran Marsupial's comments
I'm using the term hyper-parameters as Dikran Marsupial uses it in his paper (link in his answer)
I work with spectroscopic data. This is a kind of measurement where the data analysis and modelling often includes quite a bit of pre-processing. This can be seen as hyper-parameters (e.g. what order of polynomial should be used for the baseline? What measurement channels should be included?). There are other decisions that are closer to your svm parameters, e.g. how many principal components to use if a PCA is done for dimensionality reduction before the "real" model is trained? And sometimes I also use SVM classification, so I have to decide on SVM parameters.
Now, IMHO the best way to fix hyper-parameters is if you have reasons that come from the application. E.g. I usually decide on what kind of baseline to use by physical/chemical/biological reasons (i.e. knowledge about the specimen and the spectroscopic behaviour that follows from that). However, I'm not aware of such an argumentation which helps with SVM parameters...
The case of pre-experiments I mentioned above looks as follows:
- we take data of a bunch of cells (want to distinguish different cell lines).
Spectra are analysed, iterated double cross validation SVM is run (spent a night or two on the computation server).
- I observed that in the vast majoritiy of cases the same $\gamma$ and $C$ are selected as optimal. The remaining cases are a neighbour combination. This and other neighbour hyper-parameter combinations have very similar performance (one or two additional misclassifications)
- I also observe a certain overfitting: the outer cross validation isn't quite as good as the tuning results. That is as expected.
- Still, there are differences in performance over the tuning range of the hyper-parameters, and the performance over the tuning grid looks reasonably smooth. Good.
My conclusion is: while I cannot be sure that the final hyper-parameters are optimal, the outer cross validation gives me a proper estimate of the performance of the surrogate models.
During the experimental part, we decided on some changes in the experimental set-up (things that don't affect the signal to noise of the data, but that go one step further in automatization of the instrument)
We improve the experimental settings and acquire new spectra. As cells are, they need to be grown freshly. I.e. the new data set are even independent culture batches.
Now I face the decision: Should I "skip" the inner cross validation and just go with the hyper-parameters I determined with the old data?
- As mentioned above, I run the risk that these pre-determined hyper-parameters are not optimal.
- But neither can I be sure to get truly optimal hyper-parameters by doing the inner (tuning) cross validation.
- However, the tuning on the old data was stable.
- Doing the optimization I'll train on less samples: As I have anyways Too Few Samples(TM) I have to expect to obtain worse models if I set more samples aside for a second round of cross validation.
So in that case, I decided to go with fixed parameters (by experience on similar data and knowing that in the future we'll have to do our "homework" including among other things re-checking these decisions with large data).
Note that the important thing is that I skip the inner (tuning cross validation), not the outer one. With fixed hyper-parameters I get an unbiased estimate of the performance of a possibly suboptimal model. It is true that this estimate is subject to high variance, but this variance is basically the same whether I do the inner tuning or not.
Skipping the outer cross vaidation I'd get an optimistically biased estimate of a tuned model - which depending on the application and data can be worthless (if very much overoptimistic) and optimistic bias may be plainly inacceptable.