Intuition for the degrees of freedom of the LASSO

Zou et al. "On the "degrees of freedom" of the lasso" (2007) show that the number of nonzero coefficients is an unbiased and consistent estimate for the degrees of freedom of the lasso.

It seems a little counterintuitive to me.

• Suppose we have a regression model (where the variables are zero mean)

$$y=\beta x + \varepsilon.$$

• Suppose an unrestricted OLS estimate of $\beta$ is $\hat\beta_{OLS}=0.5$. It could roughly coincide with a LASSO estimate of $\beta$ for a very low penalty intensity.
• Suppose further that a LASSO estimate for a particular penalty intensity $\lambda^*$ is $\hat\beta_{LASSO,\lambda^*}=0.4$. For example, $\lambda^*$ could be the "optimal" $\lambda$ for the data set at hand found using cross validation.
• If I understand correctly, in both cases the degrees of freedom is 1 as both times there is one nonzero regression coefficient.

Question:

• How come the degrees of freedom in both cases are the same even though $\hat\beta_{LASSO,\lambda^*}=0.4$ suggests less "freedom" in fitting than $\hat\beta_{OLS}=0.5$?

References:

• great question, that would deserve more attention! – Matifou yesterday

Assume we are given a set of $n$ $p$-dimensional observations, $x_i \in \mathbb{R}^p$, $i = 1, \dotsc, n$. Assume a model of the form: \begin{align} Y_i = \langle \beta, x_i\rangle + \epsilon \end{align} where $\epsilon \sim N(0, \sigma^2)$, $\beta \in \mathbb{R}^p$, and $\langle \cdot, \cdot \rangle$ denoting the inner product. Let $\hat{\beta} = \delta(\{Y_i\}_{i=1}^n)$ be an estimate of $\beta$ using fitting method $\delta$ (either OLS or LASSO for our purposes). The formula for degrees of freedom given in the article (equation 1.2) is: \begin{align} \text{df}(\hat{\beta}) = \sum_{i=1}^n \frac{\text{Cov}(\langle\hat{\beta}, x_i\rangle, Y_i)}{\sigma^2}. \end{align}
However, when such estimates are averaged over many datasets sampled from the model, by unbiasedness and the law of large numbers such an average will converge to the true DOF. In the case of the LASSO, some of those datasets will result in an estimator wherein the coefficient is actually 0 (though such datasets might be rare if $\lambda$ is small). In the case of OLS, the estimate of the DOF is always the number of coefficients, not the number of non-zero coefficients, and so the average for the OLS case will not contain these zeros. This shows how the estimators differ, and how the average estimator for the LASSO DOF can converge to something smaller than the average estimator for the OLS DOF.
• Thanks for correcting my mistakes and imprecise formulations. Let me see if I understood you well. Essentially, if we were to repeat the experiment many times (or sample many times from the same population), we would occasionally get $\hat\beta_{LASSO}=0$ (the coefficient would be shrunk all the way to zero) and on average (across the experiments) I would get DoF for LASSO $<1$ while DoF for OLS $=1$ (obviously). – Richard Hardy Jul 11 '16 at 16:59