Why does scaling the features affect the prediction of a regression?

Question

I'm working on a regression problem using the support vector regression model from sklearn and using MinMax to scale the features, but by using it I get a different result for the regression, does that makes sense?

import pandas as pd
import numpy as np
from sklearn import  svm
from sklearn.preprocessing import MinMaxScaler

np.random.seed(0)
X_training = np.random.rand(100,15)*10
Y_training = np.random.rand(100,1)*10
model = svm.SVR()

without scaling:

model.fit(X_training,Y_training)
print model.predict(X_training)[0:10]

array([ 4.99980599,  6.99479293,  4.9784396 ,  5.03911175,  6.99557904,
        6.57214885,  6.99454049,  5.60940831,  6.99989978,  5.98628179])
Using MinMax scaler:

scaler = MinMaxScaler()
X_scaled  = scaler.fit_transform(X_training)
model.fit(X_scaled,Y_training)
model.predict(X_scaled)[0:10]

array([ 5.63521939,  6.70378514,  5.83393228,  5.33274858,  6.47539108,
        5.61135278,  5.7890052 ,  5.74425789,  6.15799404,  6.1980326 ])

Although the prediction is in the same order of magnitude there is a significant difference between both cases.

Answer 1

Regularization encompass techniques aimed at restricting model complexity. The Support Vector Machine is usually $\ell_2$-regularized, except for the intercept term, which brings coefficients asymptotically towards zero, as the cost function is amended with $\|w\|_2^2=\sum_{i=1}^pw_i^2$.

As you can see, the regularization penalty actually depends on the magnitude of the coefficients, which in turn depends on the magnitude of the features themselves. So there you have it, when you change the scale of the features you also change the scale of the coefficients, which are thus penalized differently, resulting in diverging solutions.

Answer 2

As Firebug pointed, when you scale the attributes, the regression coefficients are being generated against data whose scale is entirely different than the unscaled data. Since you used the default settings for the MinMaxScaler, your data would have taken on values between 0 and 1. Additionally, the interpretation of the coefficients changes when scaling; the scaled coefficients represent the value of the response for a unit increase of scaled value. As another example, if you had standardized your data, you would have transformed it such that each attribute was mean zero and unit standard deviation; the interpretation of a standardized attribute's impact on the response would then be: For a one unit increase/decrease in standard deviation, the corresponding response value would be "Y".

Answer 3

I did some experiments. Although I did not get the answer, here are my findings:
a) If you use the linear regression, the outputs for scaled and original data are the same.
b) If you try different values for epsilon and C parameters, the differences between the outputs for scaled and original data change. For example, here is the result for $C=10^3$:

Without Scaling: [ 4.56768351  8.26960867  2.31846237  5.03924941  9.19597982  6.57182788
  7.88097046  5.58169378  9.70512704  5.78709683]
With Scaling:    [ 4.56756895  8.47031098  2.11793562  4.83975403  9.19590282  6.57256198
  7.88051409  5.60964938  9.90450731  5.98631573]

This suggests that the way that SVR tries to solve the optimization problem is sensitive to the scaling of the data.