12
$\begingroup$

Zou et al. "On the "degrees of freedom" of the lasso" (2007) show that the number of nonzero coefficients is an unbiased and consistent estimate for the degrees of freedom of the lasso.

It seems a little counterintuitive to me.

  • Suppose we have a regression model (where the variables are zero mean)

$$y=\beta x + \varepsilon.$$

  • Suppose an unrestricted OLS estimate of $\beta$ is $\hat\beta_{OLS}=0.5$. It could roughly coincide with a LASSO estimate of $\beta$ for a very low penalty intensity.
  • Suppose further that a LASSO estimate for a particular penalty intensity $\lambda^*$ is $\hat\beta_{LASSO,\lambda^*}=0.4$. For example, $\lambda^*$ could be the "optimal" $\lambda$ for the data set at hand found using cross validation.
  • If I understand correctly, in both cases the degrees of freedom is 1 as both times there is one nonzero regression coefficient.

Question:

  • How come the degrees of freedom in both cases are the same even though $\hat\beta_{LASSO,\lambda^*}=0.4$ suggests less "freedom" in fitting than $\hat\beta_{OLS}=0.5$?

References:

$\endgroup$
  • 1
    $\begingroup$ great question, that would deserve more attention! $\endgroup$ – Matifou yesterday
8
$\begingroup$

Assume we are given a set of $n$ $p$-dimensional observations, $x_i \in \mathbb{R}^p$, $i = 1, \dotsc, n$. Assume a model of the form: \begin{align} Y_i = \langle \beta, x_i\rangle + \epsilon \end{align} where $\epsilon \sim N(0, \sigma^2)$, $\beta \in \mathbb{R}^p$, and $\langle \cdot, \cdot \rangle$ denoting the inner product. Let $\hat{\beta} = \delta(\{Y_i\}_{i=1}^n)$ be an estimate of $\beta$ using fitting method $\delta$ (either OLS or LASSO for our purposes). The formula for degrees of freedom given in the article (equation 1.2) is: \begin{align} \text{df}(\hat{\beta}) = \sum_{i=1}^n \frac{\text{Cov}(\langle\hat{\beta}, x_i\rangle, Y_i)}{\sigma^2}. \end{align}

By inspecting this formula we can surmise that, in accordance with your intuition, the true DOF for the LASSO will indeed be less than the true DOF of OLS; the coefficient-shrinkage effected by the LASSO should tend to decrease the covariances.

Now, to answer your question, the reason that the DOF for the LASSO is the same as the DOF for OLS in your example is just that there you are dealing with estimates (albeit unbiased ones), obtained from a particular dataset sampled from the model, of the true DOF values. For any particular dataset, such an estimate will not be equal to the true value (especially since the estimate is required to be an integer while the true value is a real number in general).

However, when such estimates are averaged over many datasets sampled from the model, by unbiasedness and the law of large numbers such an average will converge to the true DOF. In the case of the LASSO, some of those datasets will result in an estimator wherein the coefficient is actually 0 (though such datasets might be rare if $\lambda$ is small). In the case of OLS, the estimate of the DOF is always the number of coefficients, not the number of non-zero coefficients, and so the average for the OLS case will not contain these zeros. This shows how the estimators differ, and how the average estimator for the LASSO DOF can converge to something smaller than the average estimator for the OLS DOF.

$\endgroup$
  • 1
    $\begingroup$ Thanks for correcting my mistakes and imprecise formulations. Let me see if I understood you well. Essentially, if we were to repeat the experiment many times (or sample many times from the same population), we would occasionally get $\hat\beta_{LASSO}=0$ (the coefficient would be shrunk all the way to zero) and on average (across the experiments) I would get DoF for LASSO $<1$ while DoF for OLS $=1$ (obviously). $\endgroup$ – Richard Hardy Jul 11 '16 at 16:59
  • $\begingroup$ By the way, why does the estimate of degrees of freedom need to be integer? Does it, really? Let me also remark that the inner product notation appears unnecessarily complicated and is rarely used on this site; matrix notation would suffice. But it's your choice, of course. $\endgroup$ – Richard Hardy Jul 11 '16 at 17:07
  • 1
    $\begingroup$ Yes that about sums its up. The estimate of degrees of freedom has to be an integer for LASSO (at least for a single dataset) just because the estimate is the number of non-zero coefficients. $\endgroup$ – e2crawfo Jul 11 '16 at 17:20
  • 1
    $\begingroup$ The statement The estimate of degrees of freedom has to be an integer for LASSO just because the estimate is the number of non-zero coefficients seems highly tautological to me. In general, I don't think the df needs to be integer, from the very definition of the df you wrote. Similarly, in the ridge case, it is not necesarily zero. $\endgroup$ – Matifou May 23 at 22:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.