# Alternatives to Pre-Scaling Predictors in Lasso/Ridge Regression?

In lasso/ridge regression it's often recommended to scale predictors $$X$$ before estimation so that the coefficient estimates $$\hat{\beta}$$ will be invariant to the scale of predictors $$X$$.

Q: Is there a recommended alternative procedure such that:

1. the coefficient estimates lead to predicted values $$\hat{y}$$ which are $$X$$-scale invariant and
2. the coefficients can still be interpreted in their usual way as $$\beta_k = \frac{\partial y}{\partial x_k}$$?

The pre-scaling step gets (1) at the cost of (2). No pre-scaling step keeps (2) at the cost of (1).

Consider the standard ridge regression set-up from ISL, pg. 215.

$$\sum_{i=1}^n\left(y_i - \beta_0 - \sum_{j=1}^p\beta_j x_{ij} \right)^2 + \lambda\sum_{j=1}^p \beta_j^2$$

The pre-scaling approach would estimate:

$$\sum_{i=1}^n\left(y_i - \beta_0 - \sum_{j=1}^p\beta_j \tilde{x}_{ij} \right)^2 + \lambda\sum_{j=1}^p \beta_j^2$$

with $$\tilde{x}_{ij} = \frac{x_{ij}}{s_{x_j}}$$ and $$s_{x_j} = \sqrt{\frac{1}{n}\sum_{i=1}^n(x_{ij}-\bar{x}_{j})^2}$$

My proposal is to estimate:

$$\sum_{i=1}^n\left(y_i - \beta_0 - \sum_{j=1}^p\beta_j x_{ij} \right)^2 + \lambda\sum_{j=1}^p s_{x_j}^2 \beta_j^2$$

Note: in my proposed set-up, scaling $$x_{ij}$$ by $$c$$ with $$\check{x}_{ij}=cx_{ij}$$ gives $$s_{\check{x}_{ij}} = c s_{x_{ij}}$$. By re-parameterizing $$\check{\beta}_j = \frac{1}{c}\beta_j$$, we get back to the original objective function. In other words, the objective function and $$\hat{y}_i$$ are invariant to scalings of $$x_{ij}$$ by $$c$$.

So shouldn't my proposal get me both (1) and (2) above? Are there any bad properties I'm not seeing? Maybe there would be issues in the numerical optimization routine?

• I think the first paragraph is not right. Centering is a location shift which can only affect the intercept but not the slope. The slope can be affected by scaling, not centering. May 13, 2019 at 17:35
• @RichardHardy Good catch. You're right. I edited my question to reflect scaling rather than centering. May 13, 2019 at 17:57
• Intuition tells me that you may want to multiply by the $s_{x_j}$ rather than divide... Think of the penalty term as $\sum_{j=1}^p\lambda_j \beta_j$ where $\lambda_j = \lambda s_{x_j}$. Covariates with higher variance will require more shrinkage of the corresponding coefficient, not less. May 13, 2019 at 19:42
• @knrumsey Yeah, I think you're right. Another way to think about it: by scaling the $x_{ij}$ and taking $\check{x}_{ij}=cx_{ij}$ we get $s_{\check{x}_j}=cs_{x_j}$ and $\check{\beta}_j = \frac{1}{c}\beta_j$ producing the exact same cost, penalty, and objective function values in the setup you propose. In otherwords, scaling $x$ by $c$ should have no impact on the final $\hat{y_i}$. I'll change my original question to reflect this. May 13, 2019 at 20:28

Letting $$L$$ denote the usual likelihood function and $$\lambda$$ be a penalty factor, we maximize the penalized log likelihood given by $$\log L - \frac{1}{2} \lambda \sum_{i=1}^p (s_i\beta_i)^2$$ where $$s_1,s_2,...,s_p$$ are scale factors chosen to make $$s_i\beta_i$$ unitless. Most authors standardize the data first and do not have scale factors in the equation, but Equation 9.61 has the advantage of allowing estimation of $$\beta$$ on the original scale of the data. The usual methods (e.g., Newton–Raphson) are used to maximize 9.61.