3
$\begingroup$

As we know, Lasso regression has an objective of the form:

$\min_w \frac{||Xw - y||_2^2}{2n} + \alpha||w||_1,$

and Ridge regression has the form:

$\min_w ||Xw - y||_2^2 + \alpha||w||_2^2.$

My question is - why does the lasso regression objective contain the term $\frac{1}{2n}$ and ridge regression does not? Is there any logic in using mean squared error and not just squared error like in ridge objective?

I understand that this coefficient does not really matter - we can discard it and scale $\alpha$ to $2n\cdot \alpha$, but all the literature contains it so I am curious, if it has some reasoning.

I tried to search for the intuition of this on the internet but haven't found any explanation. Thanks in advance.

$\endgroup$
1
  • $\begingroup$ I have seen this $\frac1{2n}$ in Andrew Ng's Machine Leaning course when discussing linear regression with gradient descent rather than Lasso regularisation. I suspect the $\frac1n$ was justified as a sort of mean while the $\frac12$ was so the partial derivative of the squared term had a coefficient of $1$ and could be left out. So probably another example of machine learning reinventing statistical methods and doing it differently for no good reason. $\endgroup$
    – Henry
    Nov 22, 2022 at 23:43

1 Answer 1

3
$\begingroup$

Yours do not seem to agree with the usual definitions. For instance, James et al. (2013) An introduction to statistical learning: with applications in R, Springer (see Chap. 6) defines both the lasso and the ridge without dividing by the sample size.

Nevertheless, as you note, such a division is totally irrelevant as far as the estimates are concerned. One reason that I can think of as to why division by $n$ may make sense is when you want to cast your problem as an $M$-estimation, in which you minimize the average of your loss plus the penalty term with respect to your parameter. In this case, you might be interested in studying theoretical properties (e.g. consistency, limiting distribution, etc.). The average loss plus the penalty is typically an $\mathcal{O}(1)$, so under appropriate smoothness conditions, it will have a limiting distribution.

$\endgroup$
2
  • 1
    $\begingroup$ The textbook referenced in the answered can be downloaded (legally) from here. $\endgroup$ Nov 23, 2022 at 7:54
  • $\begingroup$ Ah, thanks @RichardHardy! It's cool they have made it accessible to everyone. Long life to open science! $\endgroup$
    – utobi
    Nov 23, 2022 at 7:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.