# What is the objective function for weighted lasso & ridge?

For weighted OLS, the objective function can be written as

$$\arg \min_{\beta} ||W^{0.5}(y - X\beta)||^2$$

This is quite similar to the objective function for plain OLS, except without the $$W$$ term:

$$\arg \min_{\beta} ||(y - X\beta)||^2$$

Now my question is how do we write the analogous weighted forms for weighted lasso and weighted ridge regression?

For unweighted ridge, the objective function is:

$$\arg \min_{\beta} ||(y - X\beta)||^2 + ||\lambda \beta||^2$$

and for lasso: $$\arg \min_{\beta} ||(y - X\beta)||^2 + ||\lambda \beta||_1$$

It's not clear to me how the weighted form should look like. I imagine the $$W$$, weights, will be applied to the first norm like it was in OLS, but what about the regularizer term?

• You can separately weight the regularization term and the OLS part. I discuss this possibility at stats.stackexchange.com/a/164546/919 in the context of Ridge regression, but similar considerations apply to Lasso.
– whuber
Commented Nov 21, 2023 at 15:39

The whole idea is that you create a trade-off. The fitting error vs the cost of increasing the complexity of the model. The weights affect the fitting error (eg either as counts or as heteroskedasticity of the datapoints). the regularisation term is on the model coefficient(s). The sum of fitting error term and regularisation term creates a trade off: one unit of fitting error is 'worth' $$\lambda$$ coefficient units. [Note you have written the objective function 'wrong' it is typically written $$\lambda \|\beta\|^2$$ ]