1
$\begingroup$

The issue: I would like to figure out what the best alpha is in a ridge regression and I get some form of bi-modal answer.

I generate a sample of 125 observations with 100 features where only 5 of them are informative.

After that, I want to fit a Ridge Regression on the data and see what the optimal value of alpha is.

I leave 25 samples aside for validation, and I use 100 samples to feed to the cross-validated Ridge regressor from scikit-learn.

from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression, RidgeCV, LarsCV, Ridge, Lasso, LassoCV


total_samples  = 125
n_samples      = 100
n_features     = 100
n_informative  = 5

total_data, total_target = make_regression(n_samples=total_samples,
                                           n_features=n_features, 
                                           n_informative=n_informative, 
                                           n_targets=1)



alphas=(1e-7, 1e-6, 1e-5, 1e-4, 1e-3, 0.01,0.05,0.1,0.5,1.0,5.0, 10.)

#this records the number of times a certain alpha is chosen
best_alpha_counter = {i:0 for i in alphas}

score_alpha = {i:[] for i in alphas}

for n_tries in np.arange(1000):

    #shuffle first, leave 25 aside to compute an out-of-sample score
    data, validation_data, target, validatation_target = train_test_split(total_data, total_target, test_size=25)


    ridge_cv = RidgeCV(cv=5, alphas=alphas, normalize=True)
    #100 samples are going into this, with a 5-fold x-validation
    ridge_cv.fit(data, target)

    best_alpha = ridge_cv.alpha_

    #record the score and the frequency of each alpha

    score_alpha[best_alpha].append(ridge_cv.score(validation_data, validatation_target))
    best_alpha_counter[best_alpha] += 1

The result is this:

    {1e-07: 315,
 1e-06: 0,
 1e-05: 0,
 0.0001: 0,
 0.001: 4,
 0.01: 23,
 0.05: 30,
 0.1: 119,
 0.5: 206,
 1.0: 271,
 5.0: 32,
 10.0: 0}

Which indicates there are two things going on. On one hand, there's a preference towards really small values of alpha, meaning towards standard OLS fitting.

On the other hand, a peak around 0.1 to 1.0 is also clearly observable.

The validation scores are:

{i: np.mean(j) for i,j in score_alpha.items()}
{1e-07: 0.98457394023171396,
 1e-06: nan,
 1e-05: nan,
 0.0001: nan,
 0.001: 0.99691191650458721,
 0.01: 0.98159477155158614,
 0.05: 0.97781304446818418,
 0.1: 0.97145807286461117,
 0.5: 0.93985612804109109,
 1.0: 0.92556178490650276,
 5.0: 0.84630997687139109,
 10.0: nan}

I'm not sure how to interpret this, since OLS actually does pretty poorly, as can be seen below:

total_samples  = 125
n_samples      = 100
n_features     = 100
n_informative  = 5

total_data, total_target = make_regression(n_samples=total_samples,
                                           n_features=n_features, 
                                           n_informative=n_informative, 
                                           n_targets=1)

scores = []
for n_tries in np.arange(1000):


    data, validation_data, target, validatation_target = train_test_split(total_data, total_target, test_size=25)

    lr = LinearRegression(normalize=True)

    cv = KFold(n_samples,n_folds=5)
    for train_idx, test_idx in cv:

        X_train, X_test, y_train, y_test = data[train_idx], data[test_idx], target[train_idx], target[test_idx]
        lr.fit(X_train, y_train)
        scores.append(lr.score(X_test, y_test))

print(np.mean(scores))

>0.769471525996

So why is Ridge Regression telling me so often that the best alpha is a very low one when the OLS result is significantly worse? Also, why do I have two behaviours in the selection of alpha, one going towards 0 and another one going towards 1?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.