5
$\begingroup$

I understand that in LASSO/Ridge it is best practice to scale covariates so that no single covariate dominates the penalized norm. However, when entering interaction terms, it is unclear whether only the constituent terms should be scaled, or the interaction should be as well. More formally,

Let be $x$ a vector of observations on a covariate and $s()$ be some scaling function, e.g., $s(x) = (x - \bar x)/\sigma_x$. Then:

Method 1: Scale constituent variables only

$\hat E(y) = \hat\beta_1s(x_1) + \hat \beta_2s(x_2) + \hat\beta_3[s(x_1) * s(x_2)]$

Method 2: Scale constituent variables, and interaction

$\hat E(y) = \hat\beta_1s(x_1) + \hat \beta_2s(x_2) + \hat\beta_3s(x_1 * x_2)$

As I see it, Method 1 has the issue that the interaction term is no longer guaranteed to have mean-zero and unit variance, and therefore its coefficient ($\hat\beta_3$) has the potential to be overly- or underly-important in the penalized norm. However, Method 2 has absolutely no interpretability. E.g., $\partial E(y)/\partial s(x_1) \neq \hat\beta_1 + \hat\beta_3 s(x_2)$ and it is not immediately clear what the correct derivative/interpretation is.

So I suppose I'm wondering what people tend to do in this scenario, and why?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

You are correct that Method 2 isn't readily interpretable. Something akin to Method 1 is called for.

Don't be trapped by the idea that all predictor coefficients need to be penalized to the same amount. Once you include binary or categorical predictors in the model, it's not even clear how to do that. You can make a choice about how much to penalize an interaction coefficient versus the individual coefficients for the predictors involved in the interaction. Software can allow for differential penalization of predictors, for example via the penalty.factor argument to the R glmnet() function. Apply your knowledge of the subject matter and your understanding of the distribution of the interaction products to make a reasonable choice.

If you are using variable selection via LASSO or elastic net, do make sure that any individual coefficients for predictors in maintained interaction terms are also kept in the model. See Example 4.3 in Statistical Learning with Sparsity.

Improve this answer
$\endgroup$
3
$\begingroup$

Scaling is the bugaboo of penalized regression. As exemplified in multiple examples here a Bayesian approach provides an intuitive way out by providing prior distributions for differences in predicted values. Though not shown there, an example related to your problem would be to have shrinkage (skeptical) priors for the inter-quartile range effect of a predictor, and another prior for a double difference, i.e., for an interaction effect. The latter is phrased as e.g. the effect of $x_1$ going from its first quartile to its 3rd quartile when $x_2$ is at its first quartile minus the $x_1$ effect when $x_2$ is at its 3rd quartile.

As @EdM pointed out, standard deviations do not apply to binary $x$. They also do not apply to asymmetrically-distributed continuous $x$. So as general scaling factors, standard deviations are problematic.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ "providing prior distributions for differences in predicted values" this is not directly clear and the provided link is a long text. Can I see this as a sort of reparameterisation where the prior probability of the parameters is computed by a transformation that considers the predictions that these parameters make? While it is an intuitive thought, the execution of such transformations sounds difficult. A concrete example would be to consider a model $y_i = \beta_0 + \beta_1 x_i$ and instead of describing a prior for the $\beta_i$ we describe a prior for the $y_i$? $\endgroup$
    – Sextus Empiricus
    Commented Dec 12 at 12:50
  • $\begingroup$ It's along those lines. I present more complicated examples here for a semiparametric model, e.g., putting a restriction on the amount of nonlinearity that a curve has in a given interval. The Stan statistical modeling system makes this all possible, supplemented by code that allows you to specify priors through contrasts. $\endgroup$
    – Frank Harrell
    Commented Dec 12 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.