Lasso cross validation

Question

I want to perform cross validation to find the regularization parameter for Lasso. I am using scikit-learn library in python. I first generate the dataset and then perform k-fold cross-validation. Here is my code (most of it from an example at scikit-learn website):

# generate some sparse data to play with
import numpy as np
n_samples, n_features = 5000, 200
X = np.random.randn(n_samples, n_features)
coef = 3 * np.random.randn(n_features)
coef[10:] = 0  # sparsify coef
y = np.dot(X, coef)

# add noise
y += 0.01 * np.random.normal((n_samples,))

# Split data in train set and test set
n_samples = X.shape[0]
X_train, y_train = X[:n_samples / 2], y[:n_samples / 2]
X_test, y_test = X[n_samples / 2:], y[n_samples / 2:]

###############################################################################
# Lasso
from sklearn.linear_model import Lasso
from sklearn.cross_validation import KFold
from matplotlib import pyplot as plt

kf = KFold(X_train.shape[0], n_folds = 10,)


alphas = np.logspace(-16, 3, num = 50, base = 2)

e_alphas = list()
e_alphas_r = list()  #holds average r2 error
for alpha in alphas:
    lasso = Lasso(alpha=alpha)
    err = list()
    err_2 = list()
    for tr_idx, tt_idx in kf:
        X_tr , X_tt = X_train[tr_idx], X_test[tt_idx]
        y_tr, y_tt = y_train[tr_idx], y_test[tt_idx]
        lasso.fit(X_tr, y_tr)
        y_hat = lasso.predict(X_tt)
        err_2.append(lasso.score(X_tt,y_tt))
        err.append(np.average((y_hat - y_tt)**2))
    e_alphas.append(np.average(err))
    e_alphas_r.append(np.average(err_2))

plt.figsize = (15,10)
fig = plt.figure()     
ax = fig.add_subplot(111)
ax.plot(alphas, e_alphas, 'b-')
ax.plot(alphas, e_alphas_r, 'g--')
ax.set_xlabel("alpha")
plt.show()

The graph of error is show in the figure at below:

I know that there are other ways in scikit-learn to do a lassoCV but I just want to know how do you select the parameter given that kind of graph I am getting. Thanks for your reply.

Answer 1

Basically you select whichever $\alpha$ gives you the lowest error rate (on a validation set). So to be complete cross-validation entails the following steps:

1. Split your data in three parts: training, validation and test.
2. Train a model with a given $\alpha$ on the train-set and test it on the validation-set and repeat this for the full range of possible $\alpha$ values in your grid.
3. Pick the best $\alpha$ value (i.e. the one that gives the lowest error)
4. Once you have complete this, retrain a new model using this optimal value of $\alpha$ on (trainset+validationset).
5. You can now evaluate your model on the test-set.

I haven't gone over your code, but your graph suggests to also look for even lower values of $\alpha$. Note, however, that learning curves usually look something like this when evaluated on the validation set:

That is, your error should be high in the beginning, drop to a low and then go somewhat up again.

The comment of @frank-harrell relates to the fact that you should probably repeat this experiment a few times to get robust estimates of your $\alpha$ values. For this you can also use k-fold cross-validation like you did so you should be fine.

Answer 2

Before going that far, run 5 bootstrap replications of the lasso procedure to make sure the features selected are stable. Otherwise your final interpretation of the lasso result will be suspect.