# Lasso cross validation

I want to perform cross validation to find the regularization parameter for Lasso. I am using scikit-learn library in python. I first generate the dataset and then perform k-fold cross-validation. Here is my code (most of it from an example at scikit-learn website):

# generate some sparse data to play with
import numpy as np
n_samples, n_features = 5000, 200
X = np.random.randn(n_samples, n_features)
coef = 3 * np.random.randn(n_features)
coef[10:] = 0  # sparsify coef
y = np.dot(X, coef)

y += 0.01 * np.random.normal((n_samples,))

# Split data in train set and test set
n_samples = X.shape[0]
X_train, y_train = X[:n_samples / 2], y[:n_samples / 2]
X_test, y_test = X[n_samples / 2:], y[n_samples / 2:]

###############################################################################
# Lasso
from sklearn.linear_model import Lasso
from sklearn.cross_validation import KFold
from matplotlib import pyplot as plt

kf = KFold(X_train.shape[0], n_folds = 10,)

alphas = np.logspace(-16, 3, num = 50, base = 2)

e_alphas = list()
e_alphas_r = list()  #holds average r2 error
for alpha in alphas:
lasso = Lasso(alpha=alpha)
err = list()
err_2 = list()
for tr_idx, tt_idx in kf:
X_tr , X_tt = X_train[tr_idx], X_test[tt_idx]
y_tr, y_tt = y_train[tr_idx], y_test[tt_idx]
lasso.fit(X_tr, y_tr)
y_hat = lasso.predict(X_tt)
err_2.append(lasso.score(X_tt,y_tt))
err.append(np.average((y_hat - y_tt)**2))
e_alphas.append(np.average(err))
e_alphas_r.append(np.average(err_2))

plt.figsize = (15,10)
fig = plt.figure()
ax.plot(alphas, e_alphas, 'b-')
ax.plot(alphas, e_alphas_r, 'g--')
ax.set_xlabel("alpha")
plt.show()


The graph of error is show in the figure at below:

I know that there are other ways in scikit-learn to do a lassoCV but I just want to know how do you select the parameter given that kind of graph I am getting. Thanks for your reply.

• If $\alpha$=0 is the best option for alpha, then that means that there is no danger of overfitting. Your unbiased estimate is the one that minimizes MSE, and essentially you have just regular OLS. – Sother Jul 2 '16 at 21:40

Basically you select whichever $\alpha$ gives you the lowest error rate (on a validation set). So to be complete cross-validation entails the following steps:

1. Split your data in three parts: training, validation and test.
2. Train a model with a given $\alpha$ on the train-set and test it on the validation-set and repeat this for the full range of possible $\alpha$ values in your grid.
3. Pick the best $\alpha$ value (i.e. the one that gives the lowest error)
4. Once you have complete this, retrain a new model using this optimal value of $\alpha$ on (trainset+validationset).
5. You can now evaluate your model on the test-set.

I haven't gone over your code, but your graph suggests to also look for even lower values of $\alpha$. Note, however, that learning curves usually look something like this when evaluated on the validation set:

That is, your error should be high in the beginning, drop to a low and then go somewhat up again.

The comment of @frank-harrell relates to the fact that you should probably repeat this experiment a few times to get robust estimates of your $\alpha$ values. For this you can also use k-fold cross-validation like you did so you should be fine.

• Split sample validation requires very large sample sizes for both training and test samples to work well, because it is volatile (results vary if split again). 100 repeats of 10-fold cross-validation can yield adequate precision (as good as bootstrap that uses fewer resamples). – Frank Harrell Sep 9 '13 at 15:18
• Most of the literature in my particular field of research (data mining) uses a set-up with k-fold cross validation. In all honesty the datasets are typically quite large though (>5K samples). As such, I share your concerns for smaller datasets, but I'm not sure where the tipping point lies. I suppose exploring the variance on the performance should give a clear answer to that for your particular dataset. – ciri Sep 9 '13 at 15:30
• This is not an issue of predictive accuracy. If you want to maximize predictive accuracy don't seem parsimony, instead using ridge regression (quadratic (L2) penalty). If you don't mind sacrificing predictive accuracy for parsimony you have to demonstrate that the parsimony is real and that the list of features is not just a random sample of features. – Frank Harrell Sep 9 '13 at 16:17
• I am doing 10-fold validation. I was also expecting a graph like the one that you have mentioned but that is exactly what my problem is. I get the graph like the one I have mentioned above. If I decrease alpha more I will find least error at that smaller alpha. So my problem is that reducing alpha seem to reduce the error as if alpha = 0 should be the best option. Also R2 fit value does not seem to vary at all with different alphas. – Shishir Pandey Sep 9 '13 at 17:58
• My guess would be that you are approximating a function that is perfectly learnable by the fit component of the Lasso procedure, since setting alpha to zero equals not regularizing. This is the same behavior as shown in the blue line of the graph that I posted (mirrored because your complexity decreases). Try playing with the noise (i.e. increase it a lot) or adding non-linearities to make the dataset 'harder'? – ciri Sep 9 '13 at 22:57

Before going that far, run 5 bootstrap replications of the lasso procedure to make sure the features selected are stable. Otherwise your final interpretation of the lasso result will be suspect.

• Could you please elaborate on that or give me some pointers? – Shishir Pandey Sep 9 '13 at 12:38
• The lasso has not been demonstrated to find the "right" variables in many cases. The reason for choosing an approach that yields parsimony instead of one that maximizes predictive accuracy (e.g., L2 norm instead of lasso's L1) is the belief that the parsimoneous solutions is "correct" in a certain sense. If you run 5 samples with replacement of size $n$ from the original sample of size $n$ and the lasso finds roughly the same non-zero coefficients each time, you can have more faith that the search for parsimony is meaningful. – Frank Harrell Sep 9 '13 at 15:20
• These points look very interesting but I do not recall it being mentioned in 'Elements of Statistical Learning'. Could you give some reference for the same? – Shishir Pandey Sep 9 '13 at 17:50
• Jelle Goeman just presented a nice talk on this at the International Society for Clinical Biostatistics conference. It should be covered far more extensively in courses and texts. Collinearity is one major cause of the problem. – Frank Harrell Sep 9 '13 at 20:01
• As I recall, when faced with, for example, two highly collinear predictor variables in a regression model, the lasso will tend to arbitrarily drop one or the other variable from the regression in addition to shrinking the coefficient of the remaining variable. A ridge regression will retain both collinear variables while shrinking their regression coefficients. – RobertF Oct 15 '14 at 14:49