# Normalization in SVM

I have applied libsvm with a linear kernel to a set of instances and I have obtained a 68 % success:

instance1 : f11, f12, f13, f14
instance2 : f21, f22, f23, f24
instance3 : f31, f32, f33, f34
instance4 : f41, f42, f43, f44
..............................
instanceN : fN1, fN2, fN3, fN4


Taking the same set of instances but multiplying each value (f11*1000 ... fN4*1000) I have obtained a 90% of success.

However, I the multiplied by 1000 is normalized the percentatge of success turn to 68%.

It seems something related with normalization but I don't know which it is the reason.

Your solution changes because you scale all features but you neglect to rescale the misclassification penalty $C$. The SVM cost function is as follows:

$$\min \|\mathbf{w}\|^2 + C \sum_i \xi_i$$

where $\mathbf{w}$ is the separating hyperplane, $\xi$ is a vector of slack variables associated to misclassification of training instances and $C$ is a parameter you must set as a user.

When you multiply all features by $1000$, $\|\mathbf{w}\|^2$ will be $1000^2$ times larger. You forgot to change $C$, which basically means that misclassifications of training instances are no longer penalized appropriately. If you rescale $C$ correctly you will end up with the same model.

If $C$ is far too small, the minimization problem becomes almost identical to minimizing $\|\mathbf{w}\|$ (which is a useless result). The result is a very simple model which isn't informative e.g. $\|\mathbf{w}\|\approx 0$.

If $C$ is far too large, the resulting model will be overly complex (e.g. $\|\mathbf{w}\|$ very large). Such a model overfits the training set and exhibits poor generalization performance. This is symptomized by high training set accuracy and poor test set accuracy (which is also a useless result).

The normalization in SVM is to subtract mean and divided by standard deviation. If you multiply your data (both training and testing) by the same ratio, after the normalization you should have still exactly the same values as the normalized original data. Are you sure you multiplied both training and testing data?

• After the normalization I have the same pecentatge of success turn to 68%. – david Oct 7 '13 at 7:59
• However, the main question is why is the reason to obtain 68% success with original values and 90% of success with the original data * 1000. – david Oct 7 '13 at 8:00
• If you don't do normalization to the original data, what success rate will you get? I suppose it has something to do with the normalization, normalization do not always improve your accuracy given different distributions of the data. – AshX Oct 8 '13 at 15:45
• Could it be some numerical effect. If your data is highly concentrated around zero for many cases, but then some part of your data is far away, maybe it results on lost of accuracy. By multiplying by 1000, you would mitigate such problem. – jpmuc Nov 4 '13 at 21:52

SVM uses Stochastic Gradeint Descent (SGD) or Gradient Descent (GD) algorithm for optimation.

If we look SGD:

$$\triangledown (j, i) = w^{(j)} + C \frac{\partial L(x_{i}, y_{i})}{\partial w^{(j)}}$$

so, it depends on input values.

It you multiply input values some big constant you increase convergence speed of the optimization problem. Therefore, you will get %90 accuracy. If you iterate enough, you will get same accuracy without multiplication of input.