Regularized linear model: adding special constraints to the coefficient

Question

I understand we can add $L_1$ or $L_2$ regularization to linear regression (Lasso and Ridge regression). In addition, it is possible to restrict the coefficient to be integers (see this post).

However, is there any related work to add special constraints to enforce the relationship between features?

• For example, suppose, I know feature 1 is much important than feature 2, so I want to make $\beta_1 \in [10,20]$ and $\beta_2 \in [1,2]$ as a constraint in the model.
• Another example would be, I think feature 1 is similar to feature 2. So, I want $|\beta_1-\beta_2|<c$.

If there is related work, please provide a link to the paper.
If not, how do people deal with incorporating domain knowledge about the importance of the feature to the model coefficient?

Answer 1

Another example would be, I think feature 1 is similar to feature 2. So, I want $|\beta_1-\beta_2|<c$.

Fusion penalty may be a relevant keyword. Tibshirani et al. (2005) suggest penalizing the differences between model coefficients. You could easily extend that to more nuanced penalizations, e.g. between multiples of coefficients or the like.

You could also see a couple of applications by Luca Barbaglia and colleagues.

References:

• Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., Knight, K., 2005. Sparsity and smoothness via the fused lasso. Journal of the Royal Statistical Society Series B 67(1), 91–108
• Barbaglia, Luca, Ines Wilms, and Christophe Croux. "Commodity dynamics: a sparse multi-class approach." Available at SSRN (2016).
• Wilms, Ines, Luca Barbaglia, and Christophe Croux. "Multi-class vector autoregressive models for multi-store sales data." KU Leuven, Faculty of Economics and Business, KBI_1617 (2016).