I am implementing linear regression using gradient descent algorithm in python. The closed form solution as well as gradient descent (without feature scaling) was giving satisfactory results. However, the moment i started using feature scaling (StandardScaler class in sklearn's preprocessing module), things have started to look a bit confusing.
I am following "Hands on Machine learning with scikit-learn & tensorflow" by Arelien Geron as well as tutorial available on http://scikit-learn.org/stable/modules/preprocessing.html
In the above references, it is clearly given that
- feature scaling is done when some of the features in the dataset are having large values compared to others
- and that feature scaling is done on the training data (x_train) and the same scaler is applied to testing data (x_test) as well so that test data is scaled the same way as training data
- No where was it mentioned that outputs need to be scaled as well. That is why, I have left y_train and y_test as unchanged
Now, taking care of the above facts, when I build a linear regressor model, the predicted values (y_predict) are far less compared to true values (y_test). In the first place, it looks like y_predict has been reduced by some factor. Am I missing out on something ?
The dataset that I am using is available at http://archive.ics.uci.edu/ml/machine-learning-databases/00291/airfoil_self_noise.dat
The first few lines of this dataset are as follows: -
800 0 0.3048 71.3 0.00266337 126.201
1000 0 0.3048 71.3 0.00266337 125.201
1250 0 0.3048 71.3 0.00266337 125.951
1600 0 0.3048 71.3 0.00266337 127.591
2000 0 0.3048 71.3 0.00266337 127.461
2500 0 0.3048 71.3 0.00266337 125.571
where the last column is output value. Clearly, the features have big difference in the values that they take(first feature is taking values in 1000s while the third feature is having values around 0.3 only). So, in my understanding, feature scaling is applicable here
now, when I build the model and compare y_test with y_predict, significant differences are obtained. First few compares as follows
y_test y_predicted
123.965 1.730859
124.835 2.659574
125.625 0.581208
123.807 0.218661
127.127 3.279522
122.724 -3.943073
126.160 4.236322
As can be seen, y_predicted is significantly smaller than y_test.
I am sharing my code snippets just in case it could help you:
scaler=preprocessing.StandardScaler().fit(x_trainn)
x_train=scaler.transform(x_trainn) #x_trainn is unscaled version of training data
x_test=scaler.transform(x_testt) #x_testt is unscaled test data
for iteration in range(n_iterations):
gradients=(2/m)*x_train.T.dot(x_train.dot(theta)-y_train)
theta=theta-eta*gradients
y_predict=x_test.dot(theta)
out=np.column_stack((y_test, y_predict))
print(pd.DataFrame(out))