I am not sure what is happening, but my cross-validaton error is always increasing with increasing alpha in ridge regression. It should technically go down and then increase.

Here is what I am doing :

n_alphas = 100
alphas = np.logspace(-3, 2, n_alphas)

Trains test split:

from sklearn import cross_validation

# Running Ridge Regression
from sklearn.metrics import mean_squared_error
coefs = np.zeros(())
ridge_tourism = linear_model.Ridge()
for a in alphas:
    for train_indices, test_indices in k_fold:
        ridge_tourism.fit(tourism_train_X[train_indices], tourism_train_Y[train_indices])  # Fitting the model
        #coefs.append(ridge_tourism.coef_) # Coeffiecients of the model


#ax.set_color_cycle(['b', 'r', 'g', 'c', 'k', 'y', 'm'])
plt.xlabel("Regularization Parameter")
plt.ylabel("Cross validation error")

It gives this:

enter image description here

Please advise


New plot of cross val error and training error of lasso with increasing alpha(regularisation parameter).

Does this graph look ok?. With increasing alpha training error would go up and now , the flexibility has reduced so, it would fit training data less proper. Also does flattening of graph makes sense?

enter image description here

  • $\begingroup$ "It should technically go down and then increase" -- why is that? $\endgroup$ – IcannotFixThis May 5 '15 at 8:12
  • $\begingroup$ I don't know. But I looked at some scikit learn plots too and there's is also increasing. Why is that? $\endgroup$ – Baktaawar May 5 '15 at 10:40
  • $\begingroup$ As $\alpha$ goes up, the weight parameters are pulled towards zeros. Therefore, that cross-validation error ultimately goes up it makes definitely sense. I'd say that a u-shaped behavior rather than what you see in your plot might depend on many things: (a) your model and its complexity (b) how you configured the cross-validation. And example: let's say you have a very simple model which is not fitting the data well, increasing $\alpha$ might only make the model's life even harder. $\endgroup$ – IcannotFixThis May 5 '15 at 11:16
  • $\begingroup$ ok pls see the new graph I have. The black is the cross validation error and red is the training error. With increasing alpha training error should increase and cross- val should decrease and then increase right. THis graph looks correct? Pls check the edit $\endgroup$ – Baktaawar May 5 '15 at 11:54
  • $\begingroup$ .. I don't see the new graph. $\endgroup$ – IcannotFixThis May 5 '15 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.