# Number of samples in scikit-Learn cost function for Ridge/Lasso regression

I am using scikit-learn to train some regression models on data and noticed that the cost function for Lasso Regression is defined like this:

$\frac{1}{2&space;n_\mathrm{samples}}\cdot&space;\left|\left|y-Xw\right|\right|_2^2&space;+&space;\alpha&space;\cdot&space;\left|\left|&space;w\right|\right|_1$ ,

whereas the cost function for e.g. Ridge Regression is shown as:

$\left|\left|y-Xw\right|\right|_2^2&space;+&space;\alpha&space;\cdot&space;\left|\left|&space;w\right|\right|_2^2$ .

I had a look in the code (Lasso & Ridge) as well and the implementations of the cost functions look like described above. I am confused why the 1/n_samples factor is only present in the Lasso regression case.

From my perspective it makes sense to have a scaling of the residuals inversely proportional to the number of samples $n_\mathrm{samples}$ so that if an algorithm is used on a dataset with more training samples the value of alpha should be somehow invariant to that. In the Elastic Net class, which can be understood as a combination of Lasso and Ridge regression, we also see that factor of 1/n_samples. Can someone explain why this factor is not present in the cost function of Ridge regression?

• The sklearn library has a github. Have you asked the maintainers why they made this choice? They would know the authoritative answer, if one exists.
– Sycorax
Commented May 18, 2022 at 13:54
• Thanks for the suggestion, I posted the question on github as well now: github.com/scikit-learn/scikit-learn/discussions/23407 Commented May 18, 2022 at 14:26
• Yeah I came across the same issue recently stackoverflow.com/questions/72260808/… I'm interested to see what the maintainers' rationale is Commented May 18, 2022 at 16:54
• Commented May 19, 2022 at 0:09

You are right that the standartization $$\gamma = \frac{c}{n}$$, where $$n$$ is the sample size, aims to make regularization term $$\alpha$$ invariant to different sample sizes. This makes sense for Lasso-like models that compute coefficients coordinate-wise via soft-thresholding: $$\mathbf{w}_j \leftarrow \mathcal{S}_{\alpha}\Big(\frac{1}{n} \langle \mathbf{x}_j, \mathbf{r}_j \rangle \Big) = \text{sign} \Big(\frac{1}{n} \langle \mathbf{x}_j, \mathbf{r}_j \rangle \Big) \Big(|\alpha| - \frac{1}{n} \langle \mathbf{x}_j, \mathbf{r}_j \rangle \Big)_+$$ Same with Elastic Net.
But in case of Ridge, which has a closed-form solution, I believe multiplication by $$\gamma$$ will be redundant and uninformative. First, let us quickly derive the solution:
$$\min_{\mathbf{w}} \big\{ \| \mathbf{y} - \mathbf{X}\mathbf{w} \|_2^2 + \alpha \| \mathbf{w} \|_2^2 \big\} \\ \frac{d}{d\mathbf{w}}\Big[ (\mathbf{y} - \mathbf{X}\mathbf{w})^T(\mathbf{y} - \mathbf{X}\mathbf{w}) + \alpha\mathbf{w}^T\mathbf{w} \Big] = \mathbf{0} \\ -2 \mathbf{X}^T(\mathbf{y} - \mathbf{X}\mathbf{w}) + 2\alpha\mathbf{w} = \mathbf{0} \\ \mathbf{X}^T(\mathbf{y} - \mathbf{X}\mathbf{w}) - \alpha\mathbf{w} = \mathbf{0} \\ \mathbf{w}^* = (\mathbf{X}^T\mathbf{X} + \alpha\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}$$ Now, what happens if we multiply the loss by $$\gamma$$? We end up with something like this: $$\mathbf{w}_{\gamma}^* = (\gamma\mathbf{X}^T\mathbf{X} + \alpha\mathbf{I})^{-1}\gamma\mathbf{X}^T\mathbf{y}$$ or using matrix factorization like SVD: $$\mathbf{w}_{\gamma}^{SVD} = \mathbf{V}(\gamma\mathbf{D}^2 + \alpha\mathbf{I})^{-1} \gamma\mathbf{D}\mathbf{U}^T\mathbf{y} \\ \mathbf{X} = \mathbf{U} \mathbf{D} \mathbf{V}^T$$ So, what does this standartization involve? Сan we explicitly state that it produces invariant results? (maybe the dual formulation will shed some light on this)
• Thank you for the answer, but I do not see your point why a multiplication by $\gamma$ would be redundant. After all, in the end the optimal set of weights has a different form than in the case without the multiplication by $\gamma$. Could you please elaborate on that? Commented May 21, 2022 at 13:10
• @Holgerillo Weights indeed will be different. My conjecture is that multiplying by $\gamma$ is useless for a closed-form solution. If you calculate $\mathbf{w}^*$ or $\mathbf{w}^*_{\gamma}$ with resampling and optimization of $\alpha$, you will likely have values of optimal $\alpha$ of different scales in both cases, while for Lasso and Elastic Net you can expect the values to be almost identical with multiplication by $\gamma$. Commented May 21, 2022 at 13:29