# Consistency of Adaptive LASSO

I'm reading the paper on Adaptive LASSO estimator (Zou, 2006). In one of the presented numerical simulation examples (Model 0 (Inconsistent lasso path), page 6 (1423)) they claim the following:

To show this numerically [read. inconsistent variable selection], we simulated 100 datasets from the foregoing model for three different combinations of sample size (n) and error variance (σ2). On each dataset, we computed the entire solution path of the lasso, then estimated the probability of the lasso solution path containing the true model.

However, I don't understand what do they mean with "then estimated the probability of the lasso solution path containing the true model". Or, in other words, how would you estimate such probability?

I can guess at least a few different ways to measure how far was the estimated values from the true values, for example:

Repeat one of the following steps for $$N$$ times:

1. Given the full LASSO solution path, mark true if for any hyperparameter $$\lambda$$ the estimated solution was $$\hat\beta = (\hat\beta_1, \hat\beta_2, \hat\beta_3 , 0)$$. That is, for some $$\lambda$$, the truly insignificant variable got estimated as $$\hat\beta_4 = 0$$, but the other three as some shrinked value (not necessarily $$\hat\beta_j = \beta_j = 5.6$$ holds). But this I understand as testing for variable selection, not true model selection

2. Given the full LASSO solution path, mark true if for any hyperparameter $$\lambda$$, the estimated solution was $$\hat\beta = (\beta_1, \beta_2, \beta_3, 0)$$ with true model parameters. This one sounds the most logical for me, but raises too many questions: firstly, no mention of threshold for the estimated parameters ($$\hat\beta_j = 5.61 = \text{ or } \neq 5.6 = \beta_j)$$, when are they said to be equal (the true model)? Secondly, unless $$\lambda$$ is close to 0, the estimates should be significantly shrinked, when compared with original true model. Lastly, LASSO surely can't achieve the true model in 50% of the cases, as they show in the paper? At least, not with my simulations..

3. The percentage of the solution path, where only those variables that are present in the true data generating process are selected. But this somewhat ties with (1.) question, and I doubt this to be the case since it would highly depend on $$\lambda$$ values.

My question: what do the authors actually mean with their presented results? I'm interested in replicating their simulation results.

EDIT: To short this down, I think I am confused by the wording "path containing the true model". My confusion comes from imagining the LASSO path, say like in this random example; their wording seems to suggest that a certain point in the graph would be the true model, which would seem to involve the estimated values, not only which are "nonzero".

On the other hand, it seems they deal with parameter estimation in other examples, so it would make sense this particular example was about variable selection.

• In your first bullet you say “variable selection, not true model selection.” What do you see as the difference between those two goals? Those terms are typically used interchangeably in the sparse regression literature. (At least, when assuming the truth is a linear model: the distinction is a bit more subtle under potential misspecification or in the single index model literature.) – mweylandt Dec 29 '18 at 11:40
• I haven’t re-read the adaptive lasso paper in a while, but I’m pretty sure the answer is your choice 1. I’ll hold off on writing up a fuller answer pending clarification (my previous comment). As you note, estimation consistency (getting the exact numerical values correct) is “measure zero” sort of thing in finite samples (hence why it’s always discussed asymptotically) so it’s not #2. Model selection consistency / variable selection consistency however, can occur in finite samples with high probability. – mweylandt Dec 29 '18 at 11:48
• @mweylandt thanks for the clarifications. My mistake here is probably seeing variable selection as a subset of model selection procedure, since with a given set of variables we can still come up with a set of potential models (differing in estimated values only) and select a final one based on some rule (relaxed lasso comes to mind here, but just as an illustration, not focusing on it here though). – runr Dec 29 '18 at 15:52
• Gotcha. Yes - in this context, “model” is used to mean a (structured) set of probability distributions (indexed by the non-zero coefficients) rather than a specific probability distribution (set of specific coefficients). The terminology is a bit overloaded, since all the sparse sub-models are technically elements of the big (all predictors) model, but it’s what the field has settled on. Adding a second modeling step (e.g., feeding lasso selected features into a random forest) makes things even more overloaded, but isn’t considered in the paper you’re looking at. – mweylandt Dec 29 '18 at 16:01
• @mweylandt but yes, I see that in every other interpretation of "model selection" it would involve a "measure zero" equality, so (1.) choice would make the most sense.(I listed it first also for the same reason). Either way, I wanted to confirm that I'm not missing anything that I haven't thought of. Again, thanks for the comments! :) – runr Dec 29 '18 at 16:01

## 1 Answer

Adaptive LASSO is used mainly for variable selection consistency. For each value of the tuning parameter in the lasso path you have a subset of variables which have nonzero coefficient estimates. If at least one of these subsets contains only the variables which truly have nonzero coefficients then it counts as the lasso solution path containing the true model - i.e. the true subset of variables is selected somewhere along the path. This is regardless of whether the estimate's value is equal to true value. So this is a measure of how accurately the correct subset of variables are selected.

Model selection is more about selecting a model by choosing values for the tuning parameters. The values of the estimates will differ depending on which model is chosen by the selected tuning parameter. The MSE of the parameters can be used to measure how accurate the values of the estimates are compared to the true value.

The adaptive lasso is supposed to have the oracle property when choosing the prediction optimal tuning parameter using, for example, cross validation. That is, it is consistent for variable selection (will include only the correct subset of variables) and model selection (will have low MSE).

• I'm marking this as answered, although similar points were discussed in the comments by @mweylandt (may be of use for future readers). Thank you both for the input! – runr Jan 25 '19 at 17:10