# My Cross-validation error is always increasing with increasing regularisation parameter

I am not sure what is happening, but my cross-validaton error is always increasing with increasing alpha in ridge regression. It should technically go down and then increase.

Here is what I am doing :

n_alphas = 100
alphas = np.logspace(-3, 2, n_alphas)


Trains test split:

from sklearn import cross_validation
k_fold=cross_validation.KFold(n=len(tourism_train_X),n_folds=5)

# Running Ridge Regression
from sklearn.metrics import mean_squared_error
mse=0.0
mse_score_ridge=[]
coefs = np.zeros(())
score=[]
ridge_tourism = linear_model.Ridge()
for a in alphas:
ridge_tourism.set_params(alpha=a)
index=0
for train_indices, test_indices in k_fold:
ridge_tourism.fit(tourism_train_X[train_indices], tourism_train_Y[train_indices])  # Fitting the model
#coefs.append(ridge_tourism.coef_) # Coeffiecients of the model
mse=mse+mean_squared_error(tourism_train_Y[test_indices],ridge_tourism.predict(tourism_train_X[test_indices]))
mse_score_ridge.append((mse/5))


Plotting:

plt.figure(figsize=(20,8))
#ax.set_color_cycle(['b', 'r', 'g', 'c', 'k', 'y', 'm'])
plt.plot(alphas,mse_score_ridge)
plt.xlabel("Regularization Parameter")
plt.ylabel("Cross validation error")


It gives this:

• As $\alpha$ goes up, the weight parameters are pulled towards zeros. Therefore, that cross-validation error ultimately goes up it makes definitely sense. I'd say that a u-shaped behavior rather than what you see in your plot might depend on many things: (a) your model and its complexity (b) how you configured the cross-validation. And example: let's say you have a very simple model which is not fitting the data well, increasing $\alpha$ might only make the model's life even harder. – IcannotFixThis May 5 '15 at 11:16