Feature scaling dramatically improves performance

Question

I am working with "Forest Coverage Type" Kaggle dataset (https://www.kaggle.com/c/forest-cover-type-prediction/data) and have applied support vector machine classification to predict forest coverage type from the predictors. After I cleaned and organized the data, I split the data into a training and testing set, and then built a svm model using the following commands in R:

library("e1071")
library(caret)
library(caTools)

set.seed(100) # for reproducibility
split = sample.split(forest_cover$Cover_Type, SplitRatio=0.7)
Forestcover_train  = subset(forest_cover, split==T)
# Test data will have the rest 30% of data
Forestcover_test  = subset(forest_cover, split==F)


forestcover_svmfit <- svm(Cover_Type ~., data=Forestcover_train, kernel='radial', gamma=0.1, cost=10, scale=F)
confusionMatrix(data=predict(forestcover_svmfit, Forestcover_train), Forestcover_train$Cover_Type)

The resulting confusion matrix is:

          Reference
Prediction    1    2    3    4    5    6    7
         1 1512    0    0    0    0    0    0
         2    0 1512    0    0    0    0    0
         3    0    0 1512    0    0    0    0
         4    0    0    0 1512    0    0    0
         5    0    0    0    0 1512    0    0
         6    0    0    0    0    0 1512    0
         7    0    0    0    0    0    0 1512

This implies that the model is perfectly predicting the coverage type for the training data. When I apply this same model to the testing data I get very different results.

svm.pred <- predict(forestcover_svmfit, Forestcover_test)
confusionMatrix(data=svm.pred, Forestcover_test$Cover_Type)

          Reference
Prediction   1   2   3   4   5   6   7
         1 647 648 648 642 646 647 648
         2   1   0   0   0   0   0   0
         3   0   0   0   0   0   0   0
         4   0   0   0   6   0   0   0
         5   0   0   0   0   2   0   0
         6   0   0   0   0   0   1   0
         7   0   0   0   0   0   0   0

This is about 14.5% accuracy, which is just slightly better than just guessing coverage_type=1 for every sample. However, when I scale the data, the results become much more palatable.

forestcover_svmfit <- svm(Cover_Type ~., data=Forestcover_train, kernel='radial', gamma=0.1, cost=10, scale=T)
confusionMatrix(data=predict(forestcover_svmfit, Forestcover_train), Forestcover_train$Cover_Type)

          Reference
Prediction    1    2    3    4    5    6    7
         1 1045  294    0    0   22    0  203
         2  267  870    0    0  147   14    0
         3    2   32  918   69   52  280    0
         4    0    2  224 1415    0  140    0
         5   57  246   57    0 1226   45    4
         6   12   55  313   28   65 1033    0
         7  129   13    0    0    0    0 1305

And applied to the testing data:

svm.pred <- predict(forestcover_svmfit, Forestcover_test)
confusionMatrix(data=svm.pred, Forestcover_test$Cover_Type)

          Reference
Prediction   1   2   3   4   5   6   7
         1 487 134   0   0   7   3  21
         2 111 427   9   0  24   5   1
         3   0  23 504  18   8  77   0
         4   0   0  30 614   0  21   0
         5  16  49  10   0 605   6   0
         6   0  13  95  16   4 536   0
         7  34   2   0   0   0   0 626

I don't understand how these results can be so different. Presumably, the training and testing sets are simply random samples from the full dataset, so it doesn't seem plausible that the model is overfitting to the extent shown here. Given that the training and testing data are on the same scales anyway, a perfect performance with the training data should imply that it will be perform relatively well on testing data, right?

---

Thanks for these responses.

I am familiar with tuning the hyperparameters, and yes, in this case, I simply chose a couple values that seemed to work well. What I found bizarre was not that the classifier was overfit, which it almost certainly is given the perfect classification performance on the training data. Instead, what I found bizarre was the combination of perfect performance on the training data, and effectively completely imperfect performance on the testing data. The perfect performance on the training data implies that the model has some separating hyperplanes that can divide the data into the different classes, and that these hyperplanes are in such a form as to perfectly separate the classes for the training data. I would expect, given an overfit model, that a perfect performance on training data would yield a high rate of misclassification. However, in this case, the misclassification is all in one direction; that is, basically all the data is being classified into a single class.

I'm wondering if there is an explanation for this, or if this is simply an instance where I should accept that scaling works because the results are better...

Answer 1

The point of hyperparameter tuning is to find a combination of $\gamma, C$ which generalize well, a compromise between over- and under-fitting the data. It seems like you've just picked two arbitrary values $\gamma, C$ and discovered that they don't generalize well. This isn't surprising, because there are lots of tuning parameter choices which are not good choices.

Overfitting is possible even if the train and test data have the same distribution. For an example of this in the case of a $k$-NN classifier, see compare the differences between Figures 2.2 and 2.3 in Elements of Statistical Learning (Hasti et al) and the supporting text.

It's possible to overfit any dataset arbitrarily well. Suppose you have a simple 1-dimensional regression problem, where the outcome is measured with some noise. You can choose a basis -- indeed, any basis that is at least as large as the number of training data -- that will interpolate all of the training data, but I'd be shocked if this model has any kind of value at predicting out-of-sample data. This is because your model is fitting the noise as well as the signal. Why do you expect an SVM to be any different?

Regarding your edit, your remark

what I found bizarre was the combination of perfect performance on the training data, and effectively completely imperfect performance on the testing data

is the textbook definition of overfitting. For example, the textbook Elements of Statistical Learning discusses this exact point throughout chapter 7, particularly pp. 228-230 which has the subheading "Optimism of the Training Error Rate."

---

Scaling the data is very common, especially in the case of SVM. Read on for an explanation of why.

Thinking about the difference in terms of the kernel function is illustrative.

An RBF kernel SVM has a single hyperparameter $\gamma$ and is usually written as

$$ k(x_1, x_2) = \exp\left(-\gamma \| x_1 - x_2 \|_2^2\right) $$

but we can rewrite this as $$ k(x_1, x_2) = \exp\left(- (x_1 - x_2)^\top (\gamma I) (x_1 - x_2)\right) $$

If wish we generalize this beyond a single scalar $\gamma$, then we can write $\Gamma$ for some P(S)D matrix.

$$ k(x_1, x_2) = \exp\left(- (x_1 - x_2)^\top \Gamma (x_1 - x_2)\right) $$ Perhaps we could simply scale each feature individually, in which case $\Gamma$ is diagonal with all non-negative entries. In this case, we might choose the diagonal elements of $\Gamma$ such that each adjusts the measurements of the features to be on a similar scale, such as $z$-scores. In this case, we've exactly recovered a kernel which uses standard deviations as a scale factor and controls bandwidth with a scalar parameter $\gamma$.

$$ k(x_1, x_2) = \exp\left(-\gamma (x_1 - x_2)^\top \begin{bmatrix} \sigma_1^{-2} & 0 & 0 & \cdots \\ 0 & \sigma_2^{-2} & 0 & \cdots \\ 0 & 0 & \sigma_3^{-2} & \cdots \\ \vdots & \vdots & \vdots & \ddots \end{bmatrix} (x_1 - x_2)\right) $$

This final kernel can be readily achieved by scaling the data by $\sigma$ first and then estimating an SVM for some values of $\gamma, C$. It's plausible because it suppresses the (arbitrary) choice of units. It's easy to see why. Suppose you have 2 features, and the first feature has values in the range $[10^{-4}, 10^{-2}]$ and the second feature has values in the range $[10^2, 10^4]$. The first feature is almost irrelevant because at either extreme, the contribution of the first feature makes only a tiny difference in the norm $\| x_1 - x_2 \|_2^2$

Of course, even if you scale your data, you'll still need to find the hyperparameters $\gamma, C$ which provide good fit to your data.

It's not always true that scaling improves results (see: scaling for SVM destroys my results), but it's a common enough finding because it tends to be the case that relevant features are present and important to the outcome.

Answer 2

Sycorax gave a pretty good overview that I wish this could be a comment.

However, I did want mention scaling also speeds up the algorithm. So you can improve performance in terms of speed within reason, of course.

All solver and optimization algorithms at their core are faster if you scale values. Since both regression and hyper parameter tuning rely on underlying optimization solvers, this speed boost carries into the larger process.