How to add self-defined regularization term in ridge regression?

Question

In typical ridge regression or (lasso in a more general sense). We have a regularization term after the MSE error term

$cost = \sum{(y_{i} - \beta x_{i})^{2}} + \lambda\sum{|\beta|^{2}}$

However, I believe in most engineering cases, there could be a-priori knowledge about the $\beta$. For instance, house price should be positively correlated with medium household income. This relationship should NOT go negative from a business logic perspective: people making too much money and house price drops, this doesn't make sense.

I am sure people must have looked Ridge/Lasso from a Bayesian perspective:

$cost = \sum{(y_{i} - \beta x_{i})^{2}} + \sum{\lambda_{i}|\beta_{i}-\hat{\beta}_{i}|^{2}}$

where $\hat{\beta}$ carries information about what $\beta$ should be.

This looks nice on paper, but in practice, $\beta$ might not be Gaussian. Using the house-price example, I only know $\beta$ is positive. So the penalty term is not smooth/differentiable. Therefore, it will be hard to tackle the problem using a gradient approach

I am wondering if there is a 'universal' approach that can solve above general-purpose regression.

Can someone share any insights here?

Answer 1

You can add extra positivity constraints. You would get several positivity constraints. Depending on the complexity of your constraints, you can solve it with an optimization algorithm, see quadratic programming and this post

Answer 2

I'll describe two broad approaches: the Bayesian approach (where prior knowledge enters in the form of the prior distribution), and the optimization approach (where prior knowledge enters in the form of constraints and/or penalties).

The regression methods you mentioned can indeed be cast in a Bayesian framework. Lasso and ridge regression correspond to the model:

$$p(y \mid x, \beta, \sigma^2) = \mathcal{N}(x \beta, \sigma^2)$$

Each of these methods is equivalent to performing MAP estimation. Lasso places a zero-mean Laplacian prior on the weights, and ridge regression uses a zero-mean Gaussian prior. The width of the prior is controlled by the regularization parameter. The loss function you wrote would be equivalent to using a Gaussian prior with mean $\hat{\beta}$.

A Bayesian approach is one example of the kind of 'universal' approach you're looking for. Do MAP estimation or go full Bayes and find the expected value of the posterior. Choose the prior based on the knowledge/assumptions you want to impose. For example, if you want to impose nonnegativity constraints, you can use a prior that's zero over negative weights. You can choose a different likelihood function if you don't think the errors are i.i.d. Gaussian, etc.

Another 'universal' approach is to add penalty terms to the loss function or constraints to the optimization problem. For example, you can impose nonnegativity constraints as @Mortezaaa pointed out:

$$\min_\beta \|y - x \beta\|^2 \quad \text{s.t. } \beta_i \ge 0 \enspace \forall i$$

It's also possible to encode more complicated kinds of assumptions. For example, the following problem imposes a smoothness penalty on the weights:

$$\min_\beta \|y - x \beta\|^2 + \lambda \sum_{i=2}^d (\beta_i - \beta_{i-1})^2$$

Increasing the penalty parameter $\lambda$ forces neighboring weights to be more similar to each other, increasing the smoothness.

You can choose the form of the the penalty/constraints to impose all kinds of structure. Many constraint forms have equivalent penalty/Lagrangian forms (and vice versa), and many of these have Bayesian equivalents (as in the case of lasso/ridge regression).

When taking this approach, it's important to be mindful that some loss functions and constraints can result in a trickier (or even intractable) problem. In these cases, it may only be possible to obtain an approximate or locally optimal solution. Some problems also require more specialized optimization algorithms. For example, ridge regression can be solved using simple convex optimization techniques, but lasso requires a more specialized solver.

Answer 3

I would just like to add that another way to take into account prior knowledge is to use either adaptive ridge regression or adaptive LASSO regression, where the lambdas with which you penalize your variables is then given by lambda*adaptive_penalty_weights where adaptive_penalty_weights=1/(abs(betas_prior)+small_epsilon)^2 (typically renormalized to sum to the nr of observations n) and where betas_prior is your prior belief of what your coefficients should be (often they are set to OLS or NNLS or regular ridge regression estimates). In Bayesian terms, adaptive LASSO regression corresponds to assuming a Laplacian prior (exponential if nonnegativity constraints are imposed), whereas adaptive ridge regression corresponds to assuming a Gaussian prior (truncated Gaussian with nonnegativity constraints). In glmnet there is the penalty.factor argument to fit such models. Glmnet also allows nonnegativity constraints to be set on particular coefficients using the lower.limits argument (as that takes a vector as input).