3
$\begingroup$

I am new to regularized regression, and I was told that Elastic net overcomes many issues of the Lasso Regression. Especially, in the case of highly correlated predictors, Lasso variable selections tend to be unstable.

I don't really understand why, therefore, I tried to demonstrate it. Hence, I created a dataset of highly, pairwise correlated variables:

# this function generate a pair wise higly correlated dataset
def generate_correlated_dataset(cov_strengh: float,n_features: int,n_samples: int) -> (np.ndarray,np.ndarray):
    # mu = 0 , centered distribution
    mu = np.zeros(n_features)

    #generating cov matrix, with cov_strengh as the covariance maximal strengh
    # allocating random values for sigma
    sigma = np.random.uniform(low=-.5,high=.5,size=(n_features,n_features))
    # ensuring 1 on diagonal and maximum strengh for variables pairwise
    for i in range(0,n_features):
        sigma[i,i] = cov_strengh
        if i < (n_features-1) and (i/2 == np.floor(i/2)):
            sigma[i,i+1] = cov_strengh
            sigma[i+1,i] = cov_strengh
    # making sigma positive definite
    sigma = np.dot(sigma , sigma.T)
    # making lambda symetrical
    sigma = sigma+sigma.T
    # generating x as N(mu,sigma)
    X = np.random.multivariate_normal(mean = mu, cov = sigma, size = n_samples)
    # random gaussian coeficients to generate y
    coef = np.random.randn(n_features)
    y = np.dot(X,coef)
    return (X,y)
# example:
X_ex,y_ex = generate_correlated_dataset(3,50,100)
pd.DataFrame(np.corrcoef(X_ex,rowvar=False)).head()

Then I use a 10-Folds CV on my dataset to compare Lasso's and Elastic Net's bias, variance, and the variance of predictors and stability. To measure stability, I measure the distance to K/2 selection, which I consider as the worst selection (random selection). Here my stability metric:

metric = 2/(n_features * K) * sum(|K/2 - nb_selection|)

I tried several values for alpha. I thought, that in those conditions Elastic net would easily have a stability dramatically higher that Lasso's. Here is my code to compare both regressions:

#generating training dataset
n_features = 50
n_samples = 100
K = 10 #number of folds
alpha = .1

# models, ratio is .5 for elastic
lasso = Lasso(alpha=alpha)
elastic = ElasticNet(alpha=alpha,l1_ratio=0.5)

# generate dataset:
X,y = generate_correlated_dataset(cov_strengh= 3,n_features= n_features,n_samples= n_samples)

# test models
bias_elastic, var_elastic, coefs_elastic = evaluate_model(X=X,y=y,K=K,model=elastic)
bias_lasso, var_lasso, coefs_lasso = evaluate_model(X=X,y=y,K=K,model=lasso)


print(f"Bias               : Elastic = {bias_elastic}, Lasso = {bias_lasso}")
print(f"Variance of the prediction: Elastic = {var_elastic}, Lasso = {var_lasso}")
print(f"Stability          : Elastic = {stability_metric(coefs_elastic)}, Lasso = {stability_metric(coefs_lasso)}")
print(f"Variance of coefficients  : Elastic = {np.mean(np.var(coefs_elastic, axis = 0))}, Lasso = {np.mean(np.var(coefs_lasso, axis = 0))}")

Evaluate model does a CV and returns the results for each folds. at the end, my results are slightly better for Elastic net, but not that much.

Bias               : Elastic = -0.03647122865037677, Lasso = -0.02128281711861078
Variance of pred: Elastic = 21.258733026121277, Lasso = 21.106866537149813
Stability          : Elastic = 0.928, Lasso = 0.892
Variance of coef  : Elastic = 0.001947616131880543, Lasso = 0.0035541923789652262

Could you show a dataset where one can see a real difference between Lasso and Elastic net in term of variable selection stability?

$\endgroup$
  • $\begingroup$ Good question. But what about the original Zou and Hastie (2005) paper proposing the elastic net? There must be some convincing examples there. Not only that; there is a simulation study there. Try looking at its design for inspiration. $\endgroup$ – Richard Hardy Nov 2 '18 at 15:39
  • $\begingroup$ I have'nt thought about their paper, I had a look to "Element of Statistical learning" ( web.stanford.edu/~hastie/ElemStatLearn ) and "An Introduction to statistical learning" ( www-bcf.usc.edu/~gareth/ISL ) and it was barely mentioned, not proven. I will read Zou and Hastie 's paper, thanks! $\endgroup$ – Théophile Pace Nov 2 '18 at 16:02
  • $\begingroup$ I read the paper (which I find surprisingly short considering it brings a new regression method) and they mention the stability of the coefficients values, and how coefficients shrink towards zero. however, I couldn't find a mention of feature selection stability. Moreover, the dataset used is really well-known and there isn't a deep analysis of the dataset. I might have missed something, but I think they were more interested in the stability when alpha is changing rather than in Kfolds CV $\endgroup$ – Théophile Pace Nov 2 '18 at 16:28
  • $\begingroup$ Then sorry for misleading you. But I hope reading the paper was not a complete waste of time. What about the simulation study in the paper, was it any useful for your purpose? $\endgroup$ – Richard Hardy Nov 2 '18 at 16:45
  • 1
    $\begingroup$ Not sure if you're still interested in the problem, but there was a paper in Journal of Machine Learning Research published a few months ago addressing your question jmlr.org/papers/volume18/17-514/17-514.pdf. It proposes a new stability metric satisfying 5 key properties of stable feature selection. They compare the LASSO and Elastic Net with regards to this metric. Associated code is in github.com/nogueirs/JMLR2018 $\endgroup$ – stats134711 Feb 13 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.