Formula for standardized Regression Coefficients(derivation and intuition)

Question

Here is the formula of standardized regression coefficients. I have two questions:

1)How do we derive this formula?

2)How can we understand intuitively this formula(I cannot understand why do we multiply old coefficient by ratio 'standard deviation of predictor/standard deviation of dependent vatiable'.

Answer 1

Basic intuition: change of units

Let's consider two models. Let's say $y$ is in units degrees.

Model 1: $x_i$ is denoted in units feet.

$$y_i = a + b x_i + \epsilon_i$$

$b$ is in units degrees per feet.

Model 2: $\tilde{x}_i = 12 x_i$, hence $\tilde{x}_i$ is denoted in units inches.

$$y_i = a + \tilde{b} \tilde{x}_i + \epsilon_i$$

$\tilde{b}$ is in units degrees per inch.

You will find that $\tilde{b} \cdot \frac{12\text{ inches}}{1\text{ foot}} = b$.

You're essentially doing the same thing with your scaling by the standard deviation. It's a change of units and linear models handle changes of units in an intuitively sensible way.

Linear transformation of data leads to linear transformation of coefficient estimates

Linear regression is an linear model. Any linear transformation of the input leads to a clearly defined linear transformation of the estimate.

Let $X$ be the $n$ by $k$ data matrix where we have $n$ observations of $k$ regressors.

Let $\tilde{X} = X A'$ so that $\tilde{\mathbf{x}}_i = A\mathbf{x}$.

Then we have:

\begin{align*} \tilde{\mathbf{b}} &= (\tilde{X}'\tilde{X})^{-1} \tilde{X}'\mathbf{y} \\ &= \left( A X'X A'\right)^{-1} AX'\mathbf{y}\\ &= A'^{-1} (X'X)^{-1} A^{-1} A X'\mathbf{y}\\ &= A'^{-1} \mathbf{b} \end{align*}

So if your data is transformed by the linear transformation $A$ so that $\tilde{\mathbf{x}}_i = A \mathbf{x}$ then your estimate $\mathbf{b}$ is transformed so that: $$\tilde{\mathbf{b}} = A'^{-1} \mathbf{b}$$

Is standardizing (i.e. subtracting mean and scaling by standard deviation) a linear transformation? No if your data does not include a constant, but yes if it does! (i.e. the first column of $X$ is a column of $1$s.)

Example: standardizing with 2 variables and a constant

Let $$ A = \begin{bmatrix} 1 & 0 & 0 \\ -\frac{\mu_1}{\sigma_1} & \frac{1}{\sigma_1} & 0 \\ -\frac{\mu_2}{\sigma_2} & 0 & \frac{1}{\sigma_2} \end{bmatrix} \quad \quad \mathbf{x} = \begin{bmatrix} 1 \\ x_1 \\ x_2 \end{bmatrix}$$

Then: $$ \tilde{\mathbf{x}} = A\mathbf{x} = \begin{bmatrix} 1 & 0 & 0 \\ -\frac{\mu_1}{\sigma_1} & \frac{1}{\sigma_1} & 0 \\ -\frac{\mu_2}{\sigma_2} & 0 & \frac{1}{\sigma_2} \end{bmatrix} \begin{bmatrix} 1 \\ x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 1 \\ \frac{x_1 - \mu_1}{\sigma_1} \\ \frac{x_2 - \mu_2}{\sigma_2} \end{bmatrix}$$

That is linear transformation $A$ will standardize your right hand side variables (i.e. subtract mean and divide by standard deviation).

You can show:

$$ A'^{-1} = \begin{bmatrix} 1 & \mu_1 & \mu_2 \\ 0 & \sigma_1 & 0 \\ 0 & 0 &\sigma_2 \end{bmatrix}$$

Hence: $$\tilde{\mathbf{b}} = A'^{-1} \mathbf{b} = \begin{bmatrix} b_0 + \mu_1 b_1 + \mu_2 b_2 \\ \sigma_1 b_1 \\ \sigma_2 b_2 \end{bmatrix}$$

You get a conceptually similar logic/result if you apply a linear transformation to $y$ (see @whuber's comment for roadmap).