I am not sure what is happening, but my cross-validaton error is always increasing with increasing alpha in ridge regression. It should technically go down and then increase.
Here is what I am doing :
n_alphas = 100 alphas = np.logspace(-3, 2, n_alphas)
Trains test split:
from sklearn import cross_validation k_fold=cross_validation.KFold(n=len(tourism_train_X),n_folds=5) # Running Ridge Regression from sklearn.metrics import mean_squared_error mse=0.0 mse_score_ridge= coefs = np.zeros(()) score= ridge_tourism = linear_model.Ridge() for a in alphas: ridge_tourism.set_params(alpha=a) index=0 for train_indices, test_indices in k_fold: ridge_tourism.fit(tourism_train_X[train_indices], tourism_train_Y[train_indices]) # Fitting the model #coefs.append(ridge_tourism.coef_) # Coeffiecients of the model mse=mse+mean_squared_error(tourism_train_Y[test_indices],ridge_tourism.predict(tourism_train_X[test_indices])) mse_score_ridge.append((mse/5))
plt.figure(figsize=(20,8)) #ax.set_color_cycle(['b', 'r', 'g', 'c', 'k', 'y', 'm']) plt.plot(alphas,mse_score_ridge) plt.xlabel("Regularization Parameter") plt.ylabel("Cross validation error")
It gives this:
New plot of cross val error and training error of lasso with increasing alpha(regularisation parameter).
Does this graph look ok?. With increasing alpha training error would go up and now , the flexibility has reduced so, it would fit training data less proper. Also does flattening of graph makes sense?