Alternatives to Pre-Scaling Predictors in Lasso/Ridge Regression?

Question

In lasso/ridge regression it's often recommended to scale predictors $X$ before estimation so that the coefficient estimates $\hat{\beta}$ will be invariant to the scale of predictors $X$.

Q: Is there a recommended alternative procedure such that:

1. the coefficient estimates lead to predicted values $\hat{y}$ which are $X$-scale invariant and
2. the coefficients can still be interpreted in their usual way as $\beta_k = \frac{\partial y}{\partial x_k}$?

The pre-scaling step gets (1) at the cost of (2). No pre-scaling step keeps (2) at the cost of (1).

What about the following approach?

Consider the standard ridge regression set-up from ISL, pg. 215.

$\sum_{i=1}^n\left(y_i - \beta_0 - \sum_{j=1}^p\beta_j x_{ij} \right)^2 + \lambda\sum_{j=1}^p \beta_j^2$

The pre-scaling approach would estimate:

$\sum_{i=1}^n\left(y_i - \beta_0 - \sum_{j=1}^p\beta_j \tilde{x}_{ij} \right)^2 + \lambda\sum_{j=1}^p \beta_j^2$

with $\tilde{x}_{ij} = \frac{x_{ij}}{s_{x_j}}$ and $s_{x_j} = \sqrt{\frac{1}{n}\sum_{i=1}^n(x_{ij}-\bar{x}_{j})^2}$

My proposal is to estimate:

$\sum_{i=1}^n\left(y_i - \beta_0 - \sum_{j=1}^p\beta_j x_{ij} \right)^2 + \lambda\sum_{j=1}^p s_{x_j}^2 \beta_j^2$

Note: in my proposed set-up, scaling $x_{ij}$ by $c$ with $\check{x}_{ij}=cx_{ij}$ gives $s_{\check{x}_{ij}} = c s_{x_{ij}}$. By re-parameterizing $\check{\beta}_j = \frac{1}{c}\beta_j$, we get back to the original objective function. In other words, the objective function and $\hat{y}_i$ are invariant to scalings of $x_{ij}$ by $c$.

So shouldn't my proposal get me both (1) and (2) above? Are there any bad properties I'm not seeing? Maybe there would be issues in the numerical optimization routine?

Answer 1

Your proposal is essentially the same as the form used by Harrell in Regression Modeling Strategies (second edition), Section 9.10, equation 9.61. He presents it in the more general context of maximum-likelihood estimation, with a log-likelihood term replacing the squared error term in your proposal:

Letting $L$ denote the usual likelihood function and $\lambda$ be a penalty factor, we maximize the penalized log likelihood given by $$ \log L - \frac{1}{2} \lambda \sum_{i=1}^p (s_i\beta_i)^2 $$ where $s_1,s_2,...,s_p$ are scale factors chosen to make $s_i\beta_i$ unitless. Most authors standardize the data first and do not have scale factors in the equation, but Equation 9.61 has the advantage of allowing estimation of $\beta$ on the original scale of the data. The usual methods (e.g., Newton–Raphson) are used to maximize 9.61.

So yes, your proposal is quite appropriate and accepted, with the advantages that you note.