Should I always pick the parameters with the best CV accuracy for GridSearch?

Question

I’m currently training an SVM for multi classification, and to choose the C and gamma parameters, I'm using Grid search combined with k-fold CV.

I get a cross validation accuracy of 99.8%. I am not quite sure of what training score means, but I suppose it’s just the accuracy you get by trying to predict the training set labels using the trained model. If this is the case, I get an accuracy of around the same percentage as the CV accuracy, but when I try to predict for my test set, I get a significantly lower accuracy (50%). I know that this means that the model is suffering from overfitting.

However, when I use a combination of parameters with which I got a little bit lower CV accuracy (around 99.2%), I get a very high training score as well as a very high test score. Which parameters should I use in this case?

Some more questions:

Does the training score need to be higher than the validation score? Does the training score should be higher than the test score? Does the validation score should be higher than the test score?

I’m a noob in machine learning, so please try not to use complex terms when answering.

Thank you in advance!

Answer 1

1. It sounds like you're doing some practice experiments so it's not that big of a deal, but in general you want to be pretty confident in your choice of hyperparameters before you even start touching the test set. Strictly, you should choose your hyperparameters only based on the training set, evaluate on the test set, and the number you get is the number you get, no backsies.

2. SVMs are mostly differentiable in their parameters, so the CV performance should be mostly smooth w.r.t. these parameters. Try plotting the CV performance against one parameter at a time and look for patterns.

3. Since you do k-fold CV, you get k performance estimates per hyperparameter value, and the 99.8 % sounds like the mean of those k values. But you might want to play it more safe than by going by the mean, e.g. looking at the mean minus a few standard deviations, or the 5th or 10th percentile of the performance estimates, or similar. This depends on the distribution of you performance estimates, how bad a bad model is considering your application, and so on.

4. The training score will usually be better than the test score, because the best hyperparameters are the best partially due to luck. This luck effect will on average disappear when you look at the test set, this is called regression toward the mean.