# confused by Ridge regression on large number of features with little number of informative ones

The issue: I would like to figure out what the best alpha is in a ridge regression and I get some form of bi-modal answer.

I generate a sample of 125 observations with 100 features where only 5 of them are informative.

After that, I want to fit a Ridge Regression on the data and see what the optimal value of alpha is.

I leave 25 samples aside for validation, and I use 100 samples to feed to the cross-validated Ridge regressor from scikit-learn.

from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression, RidgeCV, LarsCV, Ridge, Lasso, LassoCV

total_samples  = 125
n_samples      = 100
n_features     = 100
n_informative  = 5

total_data, total_target = make_regression(n_samples=total_samples,
n_features=n_features,
n_informative=n_informative,
n_targets=1)

alphas=(1e-7, 1e-6, 1e-5, 1e-4, 1e-3, 0.01,0.05,0.1,0.5,1.0,5.0, 10.)

#this records the number of times a certain alpha is chosen
best_alpha_counter = {i:0 for i in alphas}

score_alpha = {i:[] for i in alphas}

for n_tries in np.arange(1000):

#shuffle first, leave 25 aside to compute an out-of-sample score
data, validation_data, target, validatation_target = train_test_split(total_data, total_target, test_size=25)

ridge_cv = RidgeCV(cv=5, alphas=alphas, normalize=True)
#100 samples are going into this, with a 5-fold x-validation
ridge_cv.fit(data, target)

best_alpha = ridge_cv.alpha_

#record the score and the frequency of each alpha

score_alpha[best_alpha].append(ridge_cv.score(validation_data, validatation_target))
best_alpha_counter[best_alpha] += 1


The result is this:

    {1e-07: 315,
1e-06: 0,
1e-05: 0,
0.0001: 0,
0.001: 4,
0.01: 23,
0.05: 30,
0.1: 119,
0.5: 206,
1.0: 271,
5.0: 32,
10.0: 0}


Which indicates there are two things going on. On one hand, there's a preference towards really small values of alpha, meaning towards standard OLS fitting.

On the other hand, a peak around 0.1 to 1.0 is also clearly observable.

The validation scores are:

{i: np.mean(j) for i,j in score_alpha.items()}
{1e-07: 0.98457394023171396,
1e-06: nan,
1e-05: nan,
0.0001: nan,
0.001: 0.99691191650458721,
0.01: 0.98159477155158614,
0.05: 0.97781304446818418,
0.1: 0.97145807286461117,
0.5: 0.93985612804109109,
1.0: 0.92556178490650276,
5.0: 0.84630997687139109,
10.0: nan}


I'm not sure how to interpret this, since OLS actually does pretty poorly, as can be seen below:

total_samples  = 125
n_samples      = 100
n_features     = 100
n_informative  = 5

total_data, total_target = make_regression(n_samples=total_samples,
n_features=n_features,
n_informative=n_informative,
n_targets=1)

scores = []
for n_tries in np.arange(1000):

data, validation_data, target, validatation_target = train_test_split(total_data, total_target, test_size=25)

lr = LinearRegression(normalize=True)

cv = KFold(n_samples,n_folds=5)
for train_idx, test_idx in cv:

X_train, X_test, y_train, y_test = data[train_idx], data[test_idx], target[train_idx], target[test_idx]
lr.fit(X_train, y_train)
scores.append(lr.score(X_test, y_test))

print(np.mean(scores))

>0.769471525996


So why is Ridge Regression telling me so often that the best alpha is a very low one when the OLS result is significantly worse? Also, why do I have two behaviours in the selection of alpha, one going towards 0 and another one going towards 1?