4
$\begingroup$

In the second answer (https://stats.stackexchange.com/a/368426/287815) to the question (Why will ridge regression not shrink some coefficients to zero like lasso?), the OP found out that,

$β = 𝑥𝑦/(𝑥^2+λ)$

My question:

Can't the $\beta$ be 0 if one of x or y is a zero vector?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ The $x$ and $y$ are vectors (or at least collections of numbers), so it will be important to define what it means to multiply two vectors, square a vector, and add a scalar $\lambda$ to a squared vector. // Since they are vectors (or at least collections of multiple numbers), what does it mean for $x$ or $y$ to be zero? $\endgroup$
    – Dave
    Commented Aug 28, 2022 at 19:45
  • $\begingroup$ @Dave, Just edited my question: can't anyone of x or y be a zero vector? $\endgroup$
    – Deepak Tatyaji Ahire
    Commented Aug 28, 2022 at 19:47
  • $\begingroup$ $xy$ is really $\mathbf x \cdot \mathbf y$ which if zero may make linear regression not really going to work $\endgroup$
    – Henry
    Commented Aug 28, 2022 at 20:01
  • $\begingroup$ @Henry I may be wrong or I do not fully understand what you are trying to say, but y can be a zero vector, correct? And if it is, then our regression line is nothing but the X-axis. Correct me if I am wrong. $\endgroup$
    – Deepak Tatyaji Ahire
    Commented Aug 28, 2022 at 20:09
  • 2
    $\begingroup$ If all your observations of the "dependent variable" are $0$ then you may decide to always predict $0$. This is not really linear regression - you have not used the "independent variable" $\endgroup$
    – Henry
    Commented Aug 28, 2022 at 20:18

2 Answers 2

Reset to default
7
$\begingroup$

What the referred answers say is that in ridge regression the regularization does not shrink the parameters to exact zeros. It doesn't say that the parameters cannot be zeros, just that it's not what the regularization does. Your example with zero vectors is a pathological case and even without regularization, the parameter would be zero, so it has nothing to do with regularization.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hey @Tim Can you please tell me what is the difference between "shrinking to exact 0" and "parameters can be 0" mean? $\endgroup$
    – Deepak Tatyaji Ahire
    Commented Aug 28, 2022 at 20:40
  • $\begingroup$ @DeepakTatyajiAhire I mean that regression not using regularization would find the parameter equal to zero, so regularization would play no role here. $\endgroup$
    – Tim
    Commented Aug 29, 2022 at 5:06
  • $\begingroup$ I like the answer but would say something like "non-generic" or "special" rather than "pathological" ... $\endgroup$
    – Ben Bolker
    Commented Sep 1, 2022 at 19:45
1
$\begingroup$

what is the difference between "shrinking to exact 0" and "parameters can be 0"

If the OLS solution is non-zero (which means that $y$ is non-zero) then the ridge regression regularisation will not be able to shrink the parameters to exact 0.

As you found out the parameters of ridge regression can be zero when $y=0$ ($x=0$ makes no sense*). But that is only the case when $y=0$ and in that case the OLS solution is also zero (so it is not non-zero and there is nothi g to shrink). For this case it is not due to shrinking that the ridge regression parameters are zero, but due to the OLS solution being already zero.

BTW when you observe a continuous variable then the case of a zero $y$ vector and a zero coefficient will have zero probability.

* The vector $x$ won't be zero. Because that would be a useless model $y = \beta \cdot 0$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.