R e1071 tune plot does not give me the best gamma?

Question

I am using the R e1071 library for the SVM (Support Vector Machine) algorithm. And I used tune to find out the best Cost and gamma parameters.

Though the plot doesn't seem to provide the actual best prediction.

Here is some details:

gammalist <- c(0.005,0.01,0.015,0.02,0.025,0.03,0.035,0.04,0.045,0.05)
obj <- tune(svm, Class~., data = trainData, ranges = list(gamma = gammalist, cost = 2^(2:4)), tunecontrol = tune.control(sampling = "fix") )
plot(obj)

Here is the plot obtained:

The plot leads me to believe 0.02 is roundabout the best gamma. But I actually tested manually several others and find better results for 0.042.

On a 200 sample, I get 23 errors with gamma=0.042, and 26 errors with gamma=0.02.

How do you explain this?

Answer 1

When you say "better results" and provide the different errors you got, is that evaluated in-sample or out-of-sample?

SVM tuning will attempt to give you the best tuning parameters for out-of-sample prediction. But for in-sample "performance," higher gamma and higher cost will always do "better." I put 'performance' and 'better' in quotes here, b/c higher values of gamma and cost will not actually give you a better estimate of the underlying mapping between independent and outcome variables, but will "overfit" the training data. Keep in mind, SVM is capable of arbitrary complexity, so if you let it, it will perfectly fit your data. (That's assuming there are no instances of identical X's producing different Y's, but that almost surely won't happen with continuous data.) When you fit so tightly, if actually harms predictive ability, b/c there is a lot of spurious complexity in the prediction function, so my guess is that gamma=0.02 will outperform out-of-sample, while gamma=0.042 will dominate in-sample. If you're interested in prediction, go with 0.02. There aren't many reasons to go with in-sample performance, but there are instances when you want to be a little liberal with the cost and gamma parameters. (For example, if you're using SVM-regression to produce a residual which has purged variation related to all of the explanatory variables, a little over-fitting can be beneficial.) But in most cases, go with the out-of-sample prediction accuracy.

However, if you evaluated this out-of-sample and found that the higher gamma did better, then I must say that I'm surprised. It's theoretically possible for the performance manifold (the 3-d surface of out-of-sample MSE, cost and gamma) to be very spike-y, which can lead to poor tuning outcomes, but looking at the plot above, it looks pretty well-behaved.

Hope that helps!