# Naive-elastic net and elastic net variable selection comparison

The elastic net paper (here) introduced the naive-elastic net and elastic net. The coefficient estimates of naive-elastic net are obtained by solving the problem $$\hat\beta_{naive-enet}=\text{argmin}_\beta \big\lVert y - X\beta\big\rVert^2 + \lambda_1\lVert \beta\rVert_1 + \lambda_2 \lVert \beta\rVert^2_2.$$ The elastic net is a scaled version of the naive-elastic net: $$\hat\beta_{enet} = (1+\lambda_2)\hat\beta_{naive-enet}$$.

But I'm confused about the prostate data analysis in Section 4 of the paper. According to the definition above, the naive-elastic net and elastic net must select the same variables since the scaling above does not change non-zero coefficients. Hence, in the data analysis, the elastic net should have selected all the variables since the naive-elastic net included all the variables in the model. But the authors reported different selected-variables for these estimators in Table 1. How is this possible? (The picture below is from the paper.)

I know that different parameters give different models. But the elastic net is two-step process: 1. fit a naive-elastic net and 2. rescale the fitted naive-elastic net. If I get a naive-elastic net model with all the variables at the 1st step, I should get the model with the same variables in the 2nd step! It is possible that they select different variables because the parameters $$(\lambda,s)$$ are different in the two analyses. The equality $$\hat\beta_{enet}=(1+\lambda_2)\hat\beta_{naive-enet}$$ is only valid when those parameters are identical. (The parameters $$(\lambda,s)$$ are different because they are estimated through cross validation).

EDIT: adding argument for why the tuning parameters are different

Let $$\hat\beta_{enet}(X,\lambda,s)$$ and $$\hat\beta_{naive}(X,\lambda,s)$$ be the elastic net and naive elastic net estimates, respectively, based on tuning parameters $$\lambda$$ and $$s$$ and dataset $$X$$. Let $$\hat\lambda_{enet},\hat s_{enet}$$ be the chosen elastic net tuning parameters. Let $$L$$ be the loss function to compare the predicted variable $$y$$ and the predictions, $$X\hat\beta$$. Let $$X^{(1)}, \dots, X^{(10)}$$ be the 10 cross validation datasets. Then we can write

$$\hat\lambda_{enet}, \hat s_{enet} = \arg \min_{\lambda,s} \sum_{i=1}^{10} L(y,X^{(i)}\hat\beta_{enet}(\lambda,s,X^{(i)}))\\ = \arg \min_{\lambda,s} \sum_{i=1}^{10} L(y,(1+\lambda)X^{(i)}\hat\beta_{naive}(\lambda,s,X^{(i)}))\\$$

Likewise

$$\hat\lambda_{naive}, \hat s_{naive} = \arg \min_{\lambda,s} \sum_{i=1}^{10} L(y,X^{(i)}\hat\beta_{naive}(\lambda,s,X^{(i)}))\\$$

The two formulas are different, and so it is very likely that $$\hat\lambda_{enet}\neq \hat\lambda_{naive}$$ and $$\hat s_{enet}\neq \hat s_{naive}$$

• svendvn thanks for your comment. I have edited the question in line with your comment.
– mert
Jun 5 at 19:17
• Hmm.. I think my contribution gives a full answer to the question. Are there any of my statements that you disagree with? Or is it more why it answers the question? I am happy to elaborate :) Jun 5 at 20:11
• Actually, I already know the effect of the tuning parameters on the selected variables. As I mentioned in the question, I think this type of tuning parameter estimation is not suitable. I have to fit a naive-elastic net first. Hence the naive-elastic net will determine the selected variables. The elastic net is just a scaled version of the naive-elastic net. I hope I could tell what I mean.
– mert
Jun 6 at 17:40
• @mert I don't follow you completely when you say that "tuning parameter estimation is not suitable", so I have written out the tuning parameter estimation in detail. Jun 6 at 18:27