I am new to regularized regression, and I was told that Elastic net overcomes many issues of the Lasso Regression. Especially, in the case of highly correlated predictors, Lasso variable selections tend to be unstable.
I don't really understand why, therefore, I tried to demonstrate it. Hence, I created a dataset of highly, pairwise correlated variables:
# this function generate a pair wise higly correlated dataset
def generate_correlated_dataset(cov_strengh: float,n_features: int,n_samples: int) -> (np.ndarray,np.ndarray):
# mu = 0 , centered distribution
mu = np.zeros(n_features)
#generating cov matrix, with cov_strengh as the covariance maximal strengh
# allocating random values for sigma
sigma = np.random.uniform(low=-.5,high=.5,size=(n_features,n_features))
# ensuring 1 on diagonal and maximum strengh for variables pairwise
for i in range(0,n_features):
sigma[i,i] = cov_strengh
if i < (n_features-1) and (i/2 == np.floor(i/2)):
sigma[i,i+1] = cov_strengh
sigma[i+1,i] = cov_strengh
# making sigma positive definite
sigma = np.dot(sigma , sigma.T)
# making lambda symetrical
sigma = sigma+sigma.T
# generating x as N(mu,sigma)
X = np.random.multivariate_normal(mean = mu, cov = sigma, size = n_samples)
# random gaussian coeficients to generate y
coef = np.random.randn(n_features)
y = np.dot(X,coef)
return (X,y)
# example:
X_ex,y_ex = generate_correlated_dataset(3,50,100)
pd.DataFrame(np.corrcoef(X_ex,rowvar=False)).head()
Then I use a 10-Folds CV on my dataset to compare Lasso's and Elastic Net's bias, variance, and the variance of predictors and stability. To measure stability, I measure the distance to K/2 selection, which I consider as the worst selection (random selection). Here my stability metric:
metric = 2/(n_features * K) * sum(|K/2 - nb_selection|)
I tried several values for alpha. I thought, that in those conditions Elastic net would easily have a stability dramatically higher that Lasso's. Here is my code to compare both regressions:
#generating training dataset
n_features = 50
n_samples = 100
K = 10 #number of folds
alpha = .1
# models, ratio is .5 for elastic
lasso = Lasso(alpha=alpha)
elastic = ElasticNet(alpha=alpha,l1_ratio=0.5)
# generate dataset:
X,y = generate_correlated_dataset(cov_strengh= 3,n_features= n_features,n_samples= n_samples)
# test models
bias_elastic, var_elastic, coefs_elastic = evaluate_model(X=X,y=y,K=K,model=elastic)
bias_lasso, var_lasso, coefs_lasso = evaluate_model(X=X,y=y,K=K,model=lasso)
print(f"Bias : Elastic = {bias_elastic}, Lasso = {bias_lasso}")
print(f"Variance of the prediction: Elastic = {var_elastic}, Lasso = {var_lasso}")
print(f"Stability : Elastic = {stability_metric(coefs_elastic)}, Lasso = {stability_metric(coefs_lasso)}")
print(f"Variance of coefficients : Elastic = {np.mean(np.var(coefs_elastic, axis = 0))}, Lasso = {np.mean(np.var(coefs_lasso, axis = 0))}")
Evaluate model does a CV and returns the results for each folds. at the end, my results are slightly better for Elastic net, but not that much.
Bias : Elastic = -0.03647122865037677, Lasso = -0.02128281711861078
Variance of pred: Elastic = 21.258733026121277, Lasso = 21.106866537149813
Stability : Elastic = 0.928, Lasso = 0.892
Variance of coef : Elastic = 0.001947616131880543, Lasso = 0.0035541923789652262
Could you show a dataset where one can see a real difference between Lasso and Elastic net in term of variable selection stability?
