2
$\begingroup$

For both linear and logistic regression we know that the coefficient vector $\hat\beta$ holds an asymptotic normal distribution, therefore the the distribution of the linear predictor $\hat\theta_i=x^T_i\hat\beta$ is also normal.

Does the same applies for penalized regression models, such as ridge and LASSO? while I might see how $\hat\beta^{ridge}=(X^TX+\lambda I)^{-1}X^Ty$ might be proved to be normal, I'm super skeptical regarding the LASSO.

I'll be really glad if any of your answers could cite a relevant book/paper.

Improve this question
$\endgroup$
6
  • $\begingroup$ For the lasso would it not have to be some sort of discrete-continuous mix like a hurdle model since there is a decent probability that some betas are exactly zero? $\endgroup$
    – einar
    Commented Aug 31, 2023 at 6:16
  • 2
    $\begingroup$ I suspect it is still normal. maybe check out bayesian asymptotics. remember that as you get more data the regularisation has less and less impact so betas will move away from zero to the unregularised values. $\endgroup$
    – seanv507
    Commented Aug 31, 2023 at 6:53
  • 2
    $\begingroup$ I get the same sense from this paper: projecteuclid.org/journals/annals-of-statistics/volume-28/… but then again still unsure $\endgroup$
    – Spätzle
    Commented Aug 31, 2023 at 7:06
  • 3
    $\begingroup$ @einar I think we are mixing up definitions. if we start from the lagrangian formula (and using sum not mean residuals^2) and fix lambda then its clear that as N gets larger the objective is dominated by the residual term ( so the t should increase). ( I agree if we fix t and increase N this won't happen) $\endgroup$
    – seanv507
    Commented Aug 31, 2023 at 9:01
  • $\begingroup$ @Spätzle thanks for the paper. note that the assumption of $\lambda_0>0$ (asymptotic lambda as $N \rightarrow 0$) distinguishes using sum of residuals with fixed lambda vs mean of residuals with fixed lambda (which corresponds to sum of residuals with lambda multiplied by factor of N) $\endgroup$
    – seanv507
    Commented Aug 31, 2023 at 9:11

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.