2
$\begingroup$

I am working on a multiple linear regression problem with system defined as :

$Y=\sum_{j=1}^k \beta_jx_j$

I'm using the RidgeCV and LassoCV for the regression, and as mentioned in many places, the standardization of the predictor variables should be done before the Ridge or Lasso regression. Because of the physical definition of my system, I don't want to have the intercept term in the equation, that's why there is no $\beta_0$. But if I standardize the predictors with the normal way :

$z_j=\frac{x_j-\overline{x_j}}{S_j}$

when I convert the standardized betas back to origianl Bs that I want, there will be a term of intercept $\hat{\beta_0}-\sum_{j=1}^k \hat\beta_j\frac{\overline{x_j}}{S_j}$ appear as mentioned in the answer of this question Converting standardized betas back to original variables. But in my case as I said before, I don't want to have the term of intercept, how can I get my original Bs?

P.S. I'm thinking about standardize the predictors by: $z_j=\frac{x_j}{S_j}$ but I don't know if it makes any sense... I'm new to statistics so please help!

$\endgroup$
2
$\begingroup$

First, please reconsider: long experience makes us humbly suspect that even when theory indicates an intercept is unnecessary, nature may intervene in the form of errors or variation from our theory that make an intercept statistically significant and important. Leaving out the intercept can (greatly) bias all estimates, as well as providing misleading information about the uncertainties in the estimates.

But in the rare situations where an intercept definitely must be omitted, a geometric interpretation of standardization shows us what to do. Dropping the subscript $j,$ which is not needed for this discussion, your vector $\mathbf{z}$ is the result of orthogonally projecting $\mathbf{x} = (x_1,x_2, \ldots, x_n)$ onto the hyperplane orthogonal to $\mathbf{1}=(1,1,\ldots,1)$ and then rescaling that result to have a unit $L^2$ norm. Omitting the intercept means not doing the projection. Thus, the relevant analog of standardization is to rescale the unprojected vector to unit length. Accordingly, compute

$$S^2 = S(\mathbf{x})^2 = ||\mathbf{x}||^2 = \sum_{i=1}^n x_i^2$$

and then

$$\mathbf{z} = \frac{1}{S}\,\mathbf{x} = \left(\frac{x_1}{S}, \frac{x_2}{S}, \ldots, \frac{x_n}{S}\right).$$


In this context the standardized betas $\tilde\beta_i$ are, by definition, the estimates $\hat\gamma_i$ in the model

$$E[Y/S(\mathbf{y})] = \sum_i\gamma_i\, \mathbf{z}_i = \sum_i \gamma_i\,\frac{\mathbf{x}_i}{S(\mathbf{x}_i)}$$

where $\mathbf{y}$ is the vector of observed responses. Because (by virtue of the linearity of the expectation operator $E$) this is algebraically identical to the model

$$E[Y] = \sum_i \frac{S(\mathbf{y})\,\gamma_i}{S(\mathbf{x}_i)} \mathbf{x}_i = \sum_i \beta_i\, \mathbf{x}_i,$$

to convert the standardized betas into the estimates $\hat\beta_i$ for this, the original model, equate the coefficients and deduce that for each $i,$

$$\hat\beta_i = \frac{S(\mathbf{y})}{S(\mathbf{x}_i)}\,\tilde\beta_i.$$

This is recognizable as a simple change of units of measurement for $\mathbf{y}$ and $\mathbf{x}_i.$

$\endgroup$
2
  • $\begingroup$ Thank you a lot for your response! Just another question: I see that in your equations, you've standardized the dependent variable $Y$ by $Y/S(y)$ too, is it possible if I don't standardize the $Y$? Because I've seen in many places people saying that the standardization of $Y$ is not necessary... I know that the back-transformation won't be the same of cause, but I don't know if this non-standardization of Y would change the precision of the regression or something else? Thank you a lot for your time! $\endgroup$
    – Qiqi Tao
    Jun 25 '20 at 19:02
  • $\begingroup$ Not standardizing $Y$ is equivalent to setting $S(\mathbf{y})=1$ in all the equations. It's unlikely to make a difference in most calculations because the numerical precision issues arise in the analysis of the model matrix given by the $\mathbf{x}_j.$ $\endgroup$
    – whuber
    Jun 25 '20 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.