3
$\begingroup$

In lasso/ridge regression it's often recommended to scale predictors $X$ before estimation so that the coefficient estimates $\hat{\beta}$ will be invariant to the scale of predictors $X$.

Q: Is there a recommended alternative procedure such that:

  1. the coefficient estimates lead to predicted values $\hat{y}$ which are $X$-scale invariant and
  2. the coefficients can still be interpreted in their usual way as $\beta_k = \frac{\partial y}{\partial x_k}$?

The pre-scaling step gets (1) at the cost of (2). No pre-scaling step keeps (2) at the cost of (1).

What about the following approach?

Consider the standard ridge regression set-up from ISL, pg. 215.

$\sum_{i=1}^n\left(y_i - \beta_0 - \sum_{j=1}^p\beta_j x_{ij} \right)^2 + \lambda\sum_{j=1}^p \beta_j^2$

The pre-scaling approach would estimate:

$\sum_{i=1}^n\left(y_i - \beta_0 - \sum_{j=1}^p\beta_j \tilde{x}_{ij} \right)^2 + \lambda\sum_{j=1}^p \beta_j^2$

with $\tilde{x}_{ij} = \frac{x_{ij}}{s_{x_j}}$ and $s_{x_j} = \sqrt{\frac{1}{n}\sum_{i=1}^n(x_{ij}-\bar{x}_{j})^2}$

My proposal is to estimate:

$\sum_{i=1}^n\left(y_i - \beta_0 - \sum_{j=1}^p\beta_j x_{ij} \right)^2 + \lambda\sum_{j=1}^p s_{x_j}^2 \beta_j^2$

Note: in my proposed set-up, scaling $x_{ij}$ by $c$ with $\check{x}_{ij}=cx_{ij}$ gives $s_{\check{x}_{ij}} = c s_{x_{ij}}$. By re-parameterizing $\check{\beta}_j = \frac{1}{c}\beta_j$, we get back to the original objective function. In other words, the objective function and $\hat{y}_i$ are invariant to scalings of $x_{ij}$ by $c$.

So shouldn't my proposal get me both (1) and (2) above? Are there any bad properties I'm not seeing? Maybe there would be issues in the numerical optimization routine?

$\endgroup$
  • 1
    $\begingroup$ I think the first paragraph is not right. Centering is a location shift which can only affect the intercept but not the slope. The slope can be affected by scaling, not centering. $\endgroup$ – Richard Hardy May 13 at 17:35
  • $\begingroup$ @RichardHardy Good catch. You're right. I edited my question to reflect scaling rather than centering. $\endgroup$ – lowndrul May 13 at 17:57
  • 1
    $\begingroup$ Intuition tells me that you may want to multiply by the $s_{x_j}$ rather than divide... Think of the penalty term as $\sum_{j=1}^p\lambda_j \beta_j$ where $\lambda_j = \lambda s_{x_j}$. Covariates with higher variance will require more shrinkage of the corresponding coefficient, not less. $\endgroup$ – knrumsey May 13 at 19:42
  • $\begingroup$ @knrumsey Yeah, I think you're right. Another way to think about it: by scaling the $x_{ij}$ and taking $\check{x}_{ij}=cx_{ij}$ we get $s_{\check{x}_j}=cs_{x_j}$ and $\check{\beta}_j = \frac{1}{c}\beta_j$ producing the exact same cost, penalty, and objective function values in the setup you propose. In otherwords, scaling $x$ by $c$ should have no impact on the final $\hat{y_i}$. I'll change my original question to reflect this. $\endgroup$ – lowndrul May 13 at 20:28
3
$\begingroup$

Your proposal is essentially the same as the form used by Harrell in Regression Modeling Strategies (second edition), Section 9.10, equation 9.61. He presents it in the more general context of maximum-likelihood estimation, with a log-likelihood term replacing the squared error term in your proposal:

Letting $L$ denote the usual likelihood function and $\lambda$ be a penalty factor, we maximize the penalized log likelihood given by $$ \log L - \frac{1}{2} \lambda \sum_{i=1}^p (s_i\beta_i)^2 $$ where $s_1,s_2,...,s_p$ are scale factors chosen to make $s_i\beta_i$ unitless. Most authors standardize the data first and do not have scale factors in the equation, but Equation 9.61 has the advantage of allowing estimation of $\beta$ on the original scale of the data. The usual methods (e.g., Newton–Raphson) are used to maximize 9.61.

So yes, your proposal is quite appropriate and accepted, with the advantages that you note.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.