Number of samples in scikit-Learn cost function for Ridge/Lasso regression I am using scikit-learn to train some regression models on data and noticed that the cost function for Lasso Regression is defined like this:
 ,
whereas the cost function for e.g. Ridge Regression is shown as:
 .
I had a look in the code (Lasso & Ridge) as well and the implementations of the cost functions look like described above. I am confused why the 1/n_samples factor is only present in the Lasso regression case.
From my perspective it makes sense to have a scaling of the residuals inversely proportional to the number of samples  so that if an algorithm is used on a dataset with more training samples the value of alpha should be somehow invariant to that. In the Elastic Net class, which can be understood as a combination of Lasso and Ridge regression, we also see that factor of 1/n_samples. Can someone explain why this factor is not present in the cost function of Ridge regression?
 A: You are right that the standartization $\gamma = \frac{c}{n}$, where $n$ is the sample size, aims to make regularization term $\alpha$ invariant to different sample sizes. This makes sense for Lasso-like models that compute coefficients coordinate-wise via soft-thresholding:
$$
\mathbf{w}_j \leftarrow \mathcal{S}_{\alpha}\Big(\frac{1}{n} \langle \mathbf{x}_j, \mathbf{r}_j \rangle \Big) = \text{sign} \Big(\frac{1}{n} \langle \mathbf{x}_j, \mathbf{r}_j \rangle \Big) \Big(|\alpha| - \frac{1}{n} \langle \mathbf{x}_j, \mathbf{r}_j \rangle \Big)_+
$$
Same with Elastic Net.
But in case of Ridge, which has a closed-form solution, I believe multiplication by $\gamma$ will be redundant and uninformative. First, let us quickly derive the solution:
$$
\min_{\mathbf{w}} \big\{ \| \mathbf{y} - \mathbf{X}\mathbf{w} \|_2^2 + \alpha \| \mathbf{w} \|_2^2 \big\} \\
\frac{d}{d\mathbf{w}}\Big[ (\mathbf{y} - \mathbf{X}\mathbf{w})^T(\mathbf{y} - \mathbf{X}\mathbf{w}) + \alpha\mathbf{w}^T\mathbf{w} \Big] = \mathbf{0} \\
-2 \mathbf{X}^T(\mathbf{y} - \mathbf{X}\mathbf{w}) + 2\alpha\mathbf{w} = \mathbf{0} \\
\mathbf{X}^T(\mathbf{y} - \mathbf{X}\mathbf{w}) - \alpha\mathbf{w} = \mathbf{0} \\
\mathbf{w}^* = (\mathbf{X}^T\mathbf{X} + \alpha\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}
$$
Now, what happens if we multiply the loss by $\gamma$? We end up with something like this:
$$
\mathbf{w}_{\gamma}^* = (\gamma\mathbf{X}^T\mathbf{X} + \alpha\mathbf{I})^{-1}\gamma\mathbf{X}^T\mathbf{y}
$$
or using matrix factorization like SVD:
$$
\mathbf{w}_{\gamma}^{SVD} = \mathbf{V}(\gamma\mathbf{D}^2 + \alpha\mathbf{I})^{-1} \gamma\mathbf{D}\mathbf{U}^T\mathbf{y} \\
\mathbf{X} = \mathbf{U} \mathbf{D} \mathbf{V}^T
$$
So, what does this standartization involve? Сan we explicitly state that it produces invariant results? (maybe the dual formulation will shed some light on this)
