# Formula for standardized Regression Coefficients(derivation and intuition)

Here is the formula of standardized regression coefficients. I have two questions:

1)How do we derive this formula?

2)How can we understand intuitively this formula(I cannot understand why do we multiply old coefficient by ratio 'standard deviation of predictor/standard deviation of dependent vatiable'.

• It's merely a change of units of measurement. See, for instance, the analysis at stats.stackexchange.com/a/197788/919, formula $(2)$. – whuber Mar 23 '17 at 21:08
• Are you comfortable with linear algebra, matrix algebra? – Matthew Gunn Mar 23 '17 at 21:12
• @MatthewGunn Yes I am. – Daniel Yefimov Mar 23 '17 at 21:14

### Basic intuition: change of units

Let's consider two models. Let's say $y$ is in units degrees.

Model 1: $x_i$ is denoted in units feet.

$$y_i = a + b x_i + \epsilon_i$$

$b$ is in units degrees per feet.

Model 2: $\tilde{x}_i = 12 x_i$, hence $\tilde{x}_i$ is denoted in units inches.

$$y_i = a + \tilde{b} \tilde{x}_i + \epsilon_i$$

$\tilde{b}$ is in units degrees per inch.

You will find that $\tilde{b} \cdot \frac{12\text{ inches}}{1\text{ foot}} = b$.

You're essentially doing the same thing with your scaling by the standard deviation. It's a change of units and linear models handle changes of units in an intuitively sensible way.

### Linear transformation of data leads to linear transformation of coefficient estimates

Linear regression is an linear model. Any linear transformation of the input leads to a clearly defined linear transformation of the estimate.

Let $X$ be the $n$ by $k$ data matrix where we have $n$ observations of $k$ regressors.

Let $\tilde{X} = X A'$ so that $\tilde{\mathbf{x}}_i = A\mathbf{x}$.

Then we have:

\begin{align*} \tilde{\mathbf{b}} &= (\tilde{X}'\tilde{X})^{-1} \tilde{X}'\mathbf{y} \\ &= \left( A X'X A'\right)^{-1} AX'\mathbf{y}\\ &= A'^{-1} (X'X)^{-1} A^{-1} A X'\mathbf{y}\\ &= A'^{-1} \mathbf{b} \end{align*}

So if your data is transformed by the linear transformation $A$ so that $\tilde{\mathbf{x}}_i = A \mathbf{x}$ then your estimate $\mathbf{b}$ is transformed so that: $$\tilde{\mathbf{b}} = A'^{-1} \mathbf{b}$$

Is standardizing (i.e. subtracting mean and scaling by standard deviation) a linear transformation? No if your data does not include a constant, but yes if it does! (i.e. the first column of $X$ is a column of $1$s.)

### Example: standardizing with 2 variables and a constant

Let $$A = \begin{bmatrix} 1 & 0 & 0 \\ -\frac{\mu_1}{\sigma_1} & \frac{1}{\sigma_1} & 0 \\ -\frac{\mu_2}{\sigma_2} & 0 & \frac{1}{\sigma_2} \end{bmatrix} \quad \quad \mathbf{x} = \begin{bmatrix} 1 \\ x_1 \\ x_2 \end{bmatrix}$$

Then: $$\tilde{\mathbf{x}} = A\mathbf{x} = \begin{bmatrix} 1 & 0 & 0 \\ -\frac{\mu_1}{\sigma_1} & \frac{1}{\sigma_1} & 0 \\ -\frac{\mu_2}{\sigma_2} & 0 & \frac{1}{\sigma_2} \end{bmatrix} \begin{bmatrix} 1 \\ x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 1 \\ \frac{x_1 - \mu_1}{\sigma_1} \\ \frac{x_2 - \mu_2}{\sigma_2} \end{bmatrix}$$

That is linear transformation $A$ will standardize your right hand side variables (i.e. subtract mean and divide by standard deviation).

You can show:

$$A'^{-1} = \begin{bmatrix} 1 & \mu_1 & \mu_2 \\ 0 & \sigma_1 & 0 \\ 0 & 0 &\sigma_2 \end{bmatrix}$$

Hence: $$\tilde{\mathbf{b}} = A'^{-1} \mathbf{b} = \begin{bmatrix} b_0 + \mu_1 b_1 + \mu_2 b_2 \\ \sigma_1 b_1 \\ \sigma_2 b_2 \end{bmatrix}$$

You get a conceptually similar logic/result if you apply a linear transformation to $y$ (see @whuber's comment for roadmap).

• Another answer of mine on this topic: stats.stackexchange.com/questions/237039/… – Matthew Gunn Mar 23 '17 at 21:42
• +1 I thought you had in mind a more fundamental linear algebra explanation: since (when the model includes an intercept) standardization changes the design matrix $X$ into $X_0=XQ^{-1}$ for an appropriate matrix $Q$ and the augmented response matrix $Y$ into $Y_0=YR^{-1}$, then any solution $Y_0 = X\beta$ can be rewritten $Y_0R=X_0Q\beta$. Right-multiplication by $R^{-1}$ gives $Y_0 = X_0(Q\beta R^{-1})$, whence $\beta^{*}=Q\beta R^{-1}$, giving the quoted formula. (The augmented response includes a column of constants needed to subtract the mean from $Y$.) – whuber Mar 23 '17 at 21:54