0
$\begingroup$

I am not sure what is happening, but my cross-validaton error is always increasing with increasing alpha in ridge regression. It should technically go down and then increase.

Here is what I am doing :

n_alphas = 100
alphas = np.logspace(-3, 2, n_alphas)

Trains test split:

from sklearn import cross_validation
k_fold=cross_validation.KFold(n=len(tourism_train_X),n_folds=5)     

# Running Ridge Regression
from sklearn.metrics import mean_squared_error
mse=0.0
mse_score_ridge=[]
coefs = np.zeros(())
score=[]
ridge_tourism = linear_model.Ridge()
for a in alphas:
    ridge_tourism.set_params(alpha=a)
    index=0
    for train_indices, test_indices in k_fold:
        ridge_tourism.fit(tourism_train_X[train_indices], tourism_train_Y[train_indices])  # Fitting the model
        #coefs.append(ridge_tourism.coef_) # Coeffiecients of the model
        mse=mse+mean_squared_error(tourism_train_Y[test_indices],ridge_tourism.predict(tourism_train_X[test_indices]))
    mse_score_ridge.append((mse/5))

Plotting:

plt.figure(figsize=(20,8))
#ax.set_color_cycle(['b', 'r', 'g', 'c', 'k', 'y', 'm'])
plt.plot(alphas,mse_score_ridge)
plt.xlabel("Regularization Parameter")
plt.ylabel("Cross validation error")

It gives this:

enter image description here

Please advise

EDIT:

New plot of cross val error and training error of lasso with increasing alpha(regularisation parameter).

Does this graph look ok?. With increasing alpha training error would go up and now , the flexibility has reduced so, it would fit training data less proper. Also does flattening of graph makes sense?

enter image description here

$\endgroup$
  • $\begingroup$ "It should technically go down and then increase" -- why is that? $\endgroup$ – IcannotFixThis May 5 '15 at 8:12
  • $\begingroup$ I don't know. But I looked at some scikit learn plots too and there's is also increasing. Why is that? $\endgroup$ – Baktaawar May 5 '15 at 10:40
  • $\begingroup$ As $\alpha$ goes up, the weight parameters are pulled towards zeros. Therefore, that cross-validation error ultimately goes up it makes definitely sense. I'd say that a u-shaped behavior rather than what you see in your plot might depend on many things: (a) your model and its complexity (b) how you configured the cross-validation. And example: let's say you have a very simple model which is not fitting the data well, increasing $\alpha$ might only make the model's life even harder. $\endgroup$ – IcannotFixThis May 5 '15 at 11:16
  • $\begingroup$ ok pls see the new graph I have. The black is the cross validation error and red is the training error. With increasing alpha training error should increase and cross- val should decrease and then increase right. THis graph looks correct? Pls check the edit $\endgroup$ – Baktaawar May 5 '15 at 11:54
  • $\begingroup$ .. I don't see the new graph. $\endgroup$ – IcannotFixThis May 5 '15 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.