6
$\begingroup$

I am using scikit-learn to train some regression models on data and noticed that the cost function for Lasso Regression is defined like this:

formula1 ,

whereas the cost function for e.g. Ridge Regression is shown as:

formula2 .

I had a look in the code (Lasso & Ridge) as well and the implementations of the cost functions look like described above. I am confused why the 1/n_samples factor is only present in the Lasso regression case.

From my perspective it makes sense to have a scaling of the residuals inversely proportional to the number of samples inline1 so that if an algorithm is used on a dataset with more training samples the value of alpha should be somehow invariant to that. In the Elastic Net class, which can be understood as a combination of Lasso and Ridge regression, we also see that factor of 1/n_samples. Can someone explain why this factor is not present in the cost function of Ridge regression?

$\endgroup$
4

1 Answer 1

1
$\begingroup$

You are right that the standartization $\gamma = \frac{c}{n}$, where $n$ is the sample size, aims to make regularization term $\alpha$ invariant to different sample sizes. This makes sense for Lasso-like models that compute coefficients coordinate-wise via soft-thresholding: $$ \mathbf{w}_j \leftarrow \mathcal{S}_{\alpha}\Big(\frac{1}{n} \langle \mathbf{x}_j, \mathbf{r}_j \rangle \Big) = \text{sign} \Big(\frac{1}{n} \langle \mathbf{x}_j, \mathbf{r}_j \rangle \Big) \Big(|\alpha| - \frac{1}{n} \langle \mathbf{x}_j, \mathbf{r}_j \rangle \Big)_+ $$ Same with Elastic Net.

But in case of Ridge, which has a closed-form solution, I believe multiplication by $\gamma$ will be redundant and uninformative. First, let us quickly derive the solution:

$$ \min_{\mathbf{w}} \big\{ \| \mathbf{y} - \mathbf{X}\mathbf{w} \|_2^2 + \alpha \| \mathbf{w} \|_2^2 \big\} \\ \frac{d}{d\mathbf{w}}\Big[ (\mathbf{y} - \mathbf{X}\mathbf{w})^T(\mathbf{y} - \mathbf{X}\mathbf{w}) + \alpha\mathbf{w}^T\mathbf{w} \Big] = \mathbf{0} \\ -2 \mathbf{X}^T(\mathbf{y} - \mathbf{X}\mathbf{w}) + 2\alpha\mathbf{w} = \mathbf{0} \\ \mathbf{X}^T(\mathbf{y} - \mathbf{X}\mathbf{w}) - \alpha\mathbf{w} = \mathbf{0} \\ \mathbf{w}^* = (\mathbf{X}^T\mathbf{X} + \alpha\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y} $$ Now, what happens if we multiply the loss by $\gamma$? We end up with something like this: $$ \mathbf{w}_{\gamma}^* = (\gamma\mathbf{X}^T\mathbf{X} + \alpha\mathbf{I})^{-1}\gamma\mathbf{X}^T\mathbf{y} $$ or using matrix factorization like SVD: $$ \mathbf{w}_{\gamma}^{SVD} = \mathbf{V}(\gamma\mathbf{D}^2 + \alpha\mathbf{I})^{-1} \gamma\mathbf{D}\mathbf{U}^T\mathbf{y} \\ \mathbf{X} = \mathbf{U} \mathbf{D} \mathbf{V}^T $$ So, what does this standartization involve? Сan we explicitly state that it produces invariant results? (maybe the dual formulation will shed some light on this)

$\endgroup$
2
  • $\begingroup$ Thank you for the answer, but I do not see your point why a multiplication by $\gamma$ would be redundant. After all, in the end the optimal set of weights has a different form than in the case without the multiplication by $\gamma$. Could you please elaborate on that? $\endgroup$
    – Holgerillo
    Commented May 21, 2022 at 13:10
  • $\begingroup$ @Holgerillo Weights indeed will be different. My conjecture is that multiplying by $\gamma$ is useless for a closed-form solution. If you calculate $\mathbf{w}^*$ or $\mathbf{w}^*_{\gamma}$ with resampling and optimization of $\alpha$, you will likely have values of optimal $\alpha$ of different scales in both cases, while for Lasso and Elastic Net you can expect the values to be almost identical with multiplication by $\gamma$. $\endgroup$ Commented May 21, 2022 at 13:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.