0
$\begingroup$

The elastic net paper (here) introduced the naive-elastic net and elastic net. The coefficient estimates of naive-elastic net are obtained by solving the problem $$\hat\beta_{naive-enet}=\text{argmin}_\beta \big\lVert y - X\beta\big\rVert^2 + \lambda_1\lVert \beta\rVert_1 + \lambda_2 \lVert \beta\rVert^2_2.$$ The elastic net is a scaled version of the naive-elastic net: $\hat\beta_{enet} = (1+\lambda_2)\hat\beta_{naive-enet}$.

But I'm confused about the prostate data analysis in Section 4 of the paper. According to the definition above, the naive-elastic net and elastic net must select the same variables since the scaling above does not change non-zero coefficients. Hence, in the data analysis, the elastic net should have selected all the variables since the naive-elastic net included all the variables in the model. But the authors reported different selected-variables for these estimators in Table 1. How is this possible? (The picture below is from the paper.)

I know that different parameters give different models. But the elastic net is two-step process: 1. fit a naive-elastic net and 2. rescale the fitted naive-elastic net. If I get a naive-elastic net model with all the variables at the 1st step, I should get the model with the same variables in the 2nd step!

enter image description here

$\endgroup$
0
$\begingroup$

It is possible that they select different variables because the parameters $(\lambda,s)$ are different in the two analyses. The equality $\hat\beta_{enet}=(1+\lambda_2)\hat\beta_{naive-enet}$ is only valid when those parameters are identical. (The parameters $(\lambda,s)$ are different because they are estimated through cross validation).


EDIT: adding argument for why the tuning parameters are different

Let $\hat\beta_{enet}(X,\lambda,s)$ and $\hat\beta_{naive}(X,\lambda,s)$ be the elastic net and naive elastic net estimates, respectively, based on tuning parameters $\lambda$ and $s$ and dataset $X$. Let $\hat\lambda_{enet},\hat s_{enet}$ be the chosen elastic net tuning parameters. Let $L$ be the loss function to compare the predicted variable $y$ and the predictions, $X\hat\beta$. Let $X^{(1)}, \dots, X^{(10)}$ be the 10 cross validation datasets. Then we can write

$$ \hat\lambda_{enet}, \hat s_{enet} = \arg \min_{\lambda,s} \sum_{i=1}^{10} L(y,X^{(i)}\hat\beta_{enet}(\lambda,s,X^{(i)}))\\ = \arg \min_{\lambda,s} \sum_{i=1}^{10} L(y,(1+\lambda)X^{(i)}\hat\beta_{naive}(\lambda,s,X^{(i)}))\\ $$

Likewise

$$ \hat\lambda_{naive}, \hat s_{naive} = \arg \min_{\lambda,s} \sum_{i=1}^{10} L(y,X^{(i)}\hat\beta_{naive}(\lambda,s,X^{(i)}))\\ $$

The two formulas are different, and so it is very likely that $\hat\lambda_{enet}\neq \hat\lambda_{naive}$ and $\hat s_{enet}\neq \hat s_{naive}$

$\endgroup$
4
  • $\begingroup$ svendvn thanks for your comment. I have edited the question in line with your comment. $\endgroup$
    – mert
    Jun 5 at 19:17
  • $\begingroup$ Hmm.. I think my contribution gives a full answer to the question. Are there any of my statements that you disagree with? Or is it more why it answers the question? I am happy to elaborate :) $\endgroup$
    – svendvn
    Jun 5 at 20:11
  • $\begingroup$ Actually, I already know the effect of the tuning parameters on the selected variables. As I mentioned in the question, I think this type of tuning parameter estimation is not suitable. I have to fit a naive-elastic net first. Hence the naive-elastic net will determine the selected variables. The elastic net is just a scaled version of the naive-elastic net. I hope I could tell what I mean. $\endgroup$
    – mert
    Jun 6 at 17:40
  • $\begingroup$ @mert I don't follow you completely when you say that "tuning parameter estimation is not suitable", so I have written out the tuning parameter estimation in detail. $\endgroup$
    – svendvn
    Jun 6 at 18:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.