0
$\begingroup$

This question already has an answer here:

In the elements of statistical learning book the lasso is written like this: $\hat{\beta}^{lasso}=argmin\left \{ \frac{1}{2}\sum_{i=1}^{N}(y_i-\beta_0-\sum_{j=1}^{p}x_{ij}\beta_j)^2 +\lambda \sum_{j=1}^{p}|\beta_j| \right \}$ Where before the sum 1/2 is used. In some cases I see it as 1/2N. Like e.g. this equation: $\hat{\beta}^{lasso}=argmin\left \{ \frac{1}{2N}\sum_{i=1}^{N}(y_i-\sum_{j=1}^{p}x_{ij}\beta_j)^2 +\lambda \sum_{j=1}^{p}|\beta_j| \right \}$ I don't understand what this would represent if I divide with the number of observations? I would very much appreciate any claridication

$\endgroup$

marked as duplicate by kjetil b halvorsen, Michael Chernick, Sycorax, Jan Kukacka, mdewey Jun 8 '18 at 11:06

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1
$\begingroup$

It doesn't really matter from an algorithmic or solution perspective, because $\lambda$ is a "free variable" that is typically selected by cross-validation or some such. If you divide the first term in your first equation by $N$, and decrease the value of $\lambda$ by a factor of $N$, you'll have exactly the same solution.

People sometimes divide the first term by $N$ because that gives you the "Mean Squared Error" (well, $\text{MSE}/2$ in this case) which is a commonly-used metric for evaluating accuracy. If you don't divide by $N$ you get the "Sum of Squared Errors" (SSE), which varies with your sample size, which in turn makes it more difficult to compare, for example, accuracy for various sub-populations of the data that may be of different sizes.

The reason for dividing by $2$ is that when you take the derivative, the $1/2$ cancels out the "$2$" created by the exponent of the squared error term, making subsequent typing of the math slightly shorter and saving a small amount of mental effort remembering why that $2$ (that would be there if not for the cancellation) is in the subsequent equations. Again, it's irrelevant to the actual solution, as $\lambda$ can just be rescaled to match.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.