3
$\begingroup$

I understand we can add $L_1$ or $L_2$ regularization to linear regression (Lasso and Ridge regression). In addition, it is possible to restrict the coefficient to be integers (see this post).

However, is there any related work to add special constraints to enforce the relationship between features?

  • For example, suppose, I know feature 1 is much important than feature 2, so I want to make $\beta_1 \in [10,20]$ and $\beta_2 \in [1,2]$ as a constraint in the model.
  • Another example would be, I think feature 1 is similar to feature 2. So, I want $|\beta_1-\beta_2|<c$.

If there is related work, please provide a link to the paper.
If not, how do people deal with incorporating domain knowledge about the importance of the feature to the model coefficient?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ This is a standard aspect of all linear models and has nothing to do with regularization. For example, a model based on $\beta_1 x_1 + \beta_2 x_2$ will (under the constraint) instead be based on $(2\beta_2)x_1 + \beta_2 x_2 = \beta_2(2 x_1 + x_2)$ which is still a linear model with one less parameter and a new "feature" constructed from $2x_1 + x_2$. Search our site: stats.stackexchange.com/search?q=regression+constraint $\endgroup$ – whuber♦ Jul 20 '16 at 14:58
  • $\begingroup$ Thanks @whuber, I think $\beta_1=2\beta_2$ is a bad example for what I was trying to ask. I revised my question. $\endgroup$ – Haitao Du Jul 20 '16 at 15:45
2
$\begingroup$

Another example would be, I think feature 1 is similar to feature 2. So, I want $|\beta_1-\beta_2|<c$.

Fusion penalty may be a relevant keyword. Tibshirani et al. (2005) suggest penalizing the differences between model coefficients. You could easily extend that to more nuanced penalizations, e.g. between multiples of coefficients or the like.

You could also see a couple of applications by Luca Barbaglia and colleagues.

References:

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I am happy that you accepted the answer, but it could probably be better for you to wait for more answers before accepting. I only offered one example, but there could be more. $\endgroup$ – Richard Hardy Jul 22 '16 at 16:55
  • 1
    $\begingroup$ I am relatively new to CV and still exploring the level of question that can get answer. It is really really great to get answers for questions like this. Especially with good references. I like broad questions, and want more references to start to learn. this is really what I wanted. Thanks again. $\endgroup$ – Haitao Du Jul 22 '16 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.