Regression penalisation to keep correlated variables

Question

I am an avid user of logistic regression and more specifically of penalised logistic regression. With standard penalisation (Ridge, Lasso, Elasticnet) the goal is to avoid overfitting and translates with a reduction of the number of variables used. Generally it will remove strongly correlated variables.

However I am in a case where I want to keep all my variable mostly unchanged and want to make sure that similar variables play a similar role in the model. Basically if I have two explanatory variables $X_1$ and $X_2$ such that $X_1 = X_2$ for all instance, I should have $\beta_1=\beta_2$ in my calibrated model. I am unsure this is enough to fully qualify what I want to achieve, but I hope you will get the idea.

Is there any framework to achieve that trough a specific "penalisation" term ?

Answer 1

A ridge/$\ell_2$ penalty achieves what you're asking for. Consider the case of duplicate features $X_1 = X_2$ as mentioned in the question. For any choice of some value $c$, all solutions where $\beta_1 X_1 + \beta_2 X_2 = c$ will have the same cost/likelihood (assuming weights for all other features are held constant). This is why ordinary logistic regression has no unique solution in this case. But, among the continuum of equivalent choices for $\beta_1$ and $\beta_2$, the $\ell_2$ norm of the weights is minimized when $\beta_1 = \beta_2$. Thus, $\ell_2$ penalized logistic regression would set these weights to be equal.

Contrary to what the question states, an $\ell_2$ penalty does not result feature selection or sparse weights as lasso or elastic net penalties do. But, it does encourage weights to have smaller magnitude than the maximum likelihood solution (called shrinkage). If you want sparse weights, use the elastic net. The $\ell_1$ component of the elastic net penalty will encourage sparsity, while the $\ell_2$ component will encourage similar features to have similar weights (for the same reason as above).