2
$\begingroup$

I'm reading the paper on Adaptive LASSO estimator (Zou, 2006). In one of the presented numerical simulation examples (Model 0 (Inconsistent lasso path), page 6 (1423)) they claim the following:

To show this numerically [read. inconsistent variable selection], we simulated 100 datasets from the foregoing model for three different combinations of sample size (n) and error variance (σ2). On each dataset, we computed the entire solution path of the lasso, then estimated the probability of the lasso solution path containing the true model.

However, I don't understand what do they mean with "then estimated the probability of the lasso solution path containing the true model". Or, in other words, how would you estimate such probability?

I can guess at least a few different ways to measure how far was the estimated values from the true values, for example:

Repeat one of the following steps for $N$ times:

  1. Given the full LASSO solution path, mark true if for any hyperparameter $\lambda$ the estimated solution was $\hat\beta = (\hat\beta_1, \hat\beta_2, \hat\beta_3 , 0)$. That is, for some $\lambda$, the truly insignificant variable got estimated as $\hat\beta_4 = 0$, but the other three as some shrinked value (not necessarily $\hat\beta_j = \beta_j = 5.6$ holds). But this I understand as testing for variable selection, not true model selection

  2. Given the full LASSO solution path, mark true if for any hyperparameter $\lambda$, the estimated solution was $\hat\beta = (\beta_1, \beta_2, \beta_3, 0)$ with true model parameters. This one sounds the most logical for me, but raises too many questions: firstly, no mention of threshold for the estimated parameters ($\hat\beta_j = 5.61 = \text{ or } \neq 5.6 = \beta_j)$, when are they said to be equal (the true model)? Secondly, unless $\lambda$ is close to 0, the estimates should be significantly shrinked, when compared with original true model. Lastly, LASSO surely can't achieve the true model in 50% of the cases, as they show in the paper? At least, not with my simulations..

  3. The percentage of the solution path, where only those variables that are present in the true data generating process are selected. But this somewhat ties with (1.) question, and I doubt this to be the case since it would highly depend on $\lambda$ values.

My question: what do the authors actually mean with their presented results? I'm interested in replicating their simulation results.

EDIT: To short this down, I think I am confused by the wording "path containing the true model". My confusion comes from imagining the LASSO path, say like in this random example; their wording seems to suggest that a certain point in the graph would be the true model, which would seem to involve the estimated values, not only which are "nonzero".

On the other hand, it seems they deal with parameter estimation in other examples, so it would make sense this particular example was about variable selection.

$\endgroup$
  • $\begingroup$ In your first bullet you say “variable selection, not true model selection.” What do you see as the difference between those two goals? Those terms are typically used interchangeably in the sparse regression literature. (At least, when assuming the truth is a linear model: the distinction is a bit more subtle under potential misspecification or in the single index model literature.) $\endgroup$ – mweylandt Dec 29 '18 at 11:40
  • $\begingroup$ I haven’t re-read the adaptive lasso paper in a while, but I’m pretty sure the answer is your choice 1. I’ll hold off on writing up a fuller answer pending clarification (my previous comment). As you note, estimation consistency (getting the exact numerical values correct) is “measure zero” sort of thing in finite samples (hence why it’s always discussed asymptotically) so it’s not #2. Model selection consistency / variable selection consistency however, can occur in finite samples with high probability. $\endgroup$ – mweylandt Dec 29 '18 at 11:48
  • $\begingroup$ @mweylandt thanks for the clarifications. My mistake here is probably seeing variable selection as a subset of model selection procedure, since with a given set of variables we can still come up with a set of potential models (differing in estimated values only) and select a final one based on some rule (relaxed lasso comes to mind here, but just as an illustration, not focusing on it here though). $\endgroup$ – Nutle Dec 29 '18 at 15:52
  • $\begingroup$ Gotcha. Yes - in this context, “model” is used to mean a (structured) set of probability distributions (indexed by the non-zero coefficients) rather than a specific probability distribution (set of specific coefficients). The terminology is a bit overloaded, since all the sparse sub-models are technically elements of the big (all predictors) model, but it’s what the field has settled on. Adding a second modeling step (e.g., feeding lasso selected features into a random forest) makes things even more overloaded, but isn’t considered in the paper you’re looking at. $\endgroup$ – mweylandt Dec 29 '18 at 16:01
  • $\begingroup$ @mweylandt but yes, I see that in every other interpretation of "model selection" it would involve a "measure zero" equality, so (1.) choice would make the most sense.(I listed it first also for the same reason). Either way, I wanted to confirm that I'm not missing anything that I haven't thought of. Again, thanks for the comments! :) $\endgroup$ – Nutle Dec 29 '18 at 16:01

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.