5
$\begingroup$

enter image description here

Here is the formula of standardized regression coefficients. I have two questions:

1)How do we derive this formula?

2)How can we understand intuitively this formula(I cannot understand why do we multiply old coefficient by ratio 'standard deviation of predictor/standard deviation of dependent vatiable'.

$\endgroup$
3
  • 2
    $\begingroup$ It's merely a change of units of measurement. See, for instance, the analysis at stats.stackexchange.com/a/197788/919, formula $(2)$. $\endgroup$
    – whuber
    Mar 23, 2017 at 21:08
  • $\begingroup$ Are you comfortable with linear algebra, matrix algebra? $\endgroup$ Mar 23, 2017 at 21:12
  • $\begingroup$ @MatthewGunn Yes I am. $\endgroup$ Mar 23, 2017 at 21:14

1 Answer 1

5
$\begingroup$

Basic intuition: change of units

Let's consider two models. Let's say $y$ is in units degrees.

Model 1: $x_i$ is denoted in units feet.

$$y_i = a + b x_i + \epsilon_i$$

$b$ is in units degrees per feet.

Model 2: $\tilde{x}_i = 12 x_i$, hence $\tilde{x}_i$ is denoted in units inches.

$$y_i = a + \tilde{b} \tilde{x}_i + \epsilon_i$$

$\tilde{b}$ is in units degrees per inch.

You will find that $\tilde{b} \cdot \frac{12\text{ inches}}{1\text{ foot}} = b$.

You're essentially doing the same thing with your scaling by the standard deviation. It's a change of units and linear models handle changes of units in an intuitively sensible way.

Linear transformation of data leads to linear transformation of coefficient estimates

Linear regression is an linear model. Any linear transformation of the input leads to a clearly defined linear transformation of the estimate.

Let $X$ be the $n$ by $k$ data matrix where we have $n$ observations of $k$ regressors.

Let $\tilde{X} = X A'$ so that $\tilde{\mathbf{x}}_i = A\mathbf{x}$.

Then we have:

\begin{align*} \tilde{\mathbf{b}} &= (\tilde{X}'\tilde{X})^{-1} \tilde{X}'\mathbf{y} \\ &= \left( A X'X A'\right)^{-1} AX'\mathbf{y}\\ &= A'^{-1} (X'X)^{-1} A^{-1} A X'\mathbf{y}\\ &= A'^{-1} \mathbf{b} \end{align*}

So if your data is transformed by the linear transformation $A$ so that $\tilde{\mathbf{x}}_i = A \mathbf{x}$ then your estimate $\mathbf{b}$ is transformed so that: $$\tilde{\mathbf{b}} = A'^{-1} \mathbf{b}$$

Is standardizing (i.e. subtracting mean and scaling by standard deviation) a linear transformation? No if your data does not include a constant, but yes if it does! (i.e. the first column of $X$ is a column of $1$s.)

Example: standardizing with 2 variables and a constant

Let $$ A = \begin{bmatrix} 1 & 0 & 0 \\ -\frac{\mu_1}{\sigma_1} & \frac{1}{\sigma_1} & 0 \\ -\frac{\mu_2}{\sigma_2} & 0 & \frac{1}{\sigma_2} \end{bmatrix} \quad \quad \mathbf{x} = \begin{bmatrix} 1 \\ x_1 \\ x_2 \end{bmatrix}$$

Then: $$ \tilde{\mathbf{x}} = A\mathbf{x} = \begin{bmatrix} 1 & 0 & 0 \\ -\frac{\mu_1}{\sigma_1} & \frac{1}{\sigma_1} & 0 \\ -\frac{\mu_2}{\sigma_2} & 0 & \frac{1}{\sigma_2} \end{bmatrix} \begin{bmatrix} 1 \\ x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 1 \\ \frac{x_1 - \mu_1}{\sigma_1} \\ \frac{x_2 - \mu_2}{\sigma_2} \end{bmatrix}$$

That is linear transformation $A$ will standardize your right hand side variables (i.e. subtract mean and divide by standard deviation).

You can show:

$$ A'^{-1} = \begin{bmatrix} 1 & \mu_1 & \mu_2 \\ 0 & \sigma_1 & 0 \\ 0 & 0 &\sigma_2 \end{bmatrix}$$

Hence: $$\tilde{\mathbf{b}} = A'^{-1} \mathbf{b} = \begin{bmatrix} b_0 + \mu_1 b_1 + \mu_2 b_2 \\ \sigma_1 b_1 \\ \sigma_2 b_2 \end{bmatrix}$$

You get a conceptually similar logic/result if you apply a linear transformation to $y$ (see @whuber's comment for roadmap).

$\endgroup$
2
  • $\begingroup$ Another answer of mine on this topic: stats.stackexchange.com/questions/237039/… $\endgroup$ Mar 23, 2017 at 21:42
  • 1
    $\begingroup$ +1 I thought you had in mind a more fundamental linear algebra explanation: since (when the model includes an intercept) standardization changes the design matrix $X$ into $X_0=XQ^{-1}$ for an appropriate matrix $Q$ and the augmented response matrix $Y$ into $Y_0=YR^{-1}$, then any solution $Y_0 = X\beta$ can be rewritten $Y_0R=X_0Q\beta$. Right-multiplication by $R^{-1}$ gives $Y_0 = X_0(Q\beta R^{-1})$, whence $\beta^{*}=Q\beta R^{-1}$, giving the quoted formula. (The augmented response includes a column of constants needed to subtract the mean from $Y$.) $\endgroup$
    – whuber
    Mar 23, 2017 at 21:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.