Why does the sign of Regression coefficients for standardized & unstandardize data using R differ

Question

I'm doing a liner regression fit using R. I used lm() to do the regression. Then I standardize my data using scale() and again do the regression on standardize data using lm().

Surprisingly the regression coefficient of one variable was positive before standardization and after standardization I found it is showing negative coefficient. I checked the correlation between that variable and my predictor. It has positive significant correlation.

data_bd2=data_2[,c(1:3,5:7)] 
str(data_bd2) 
fit_bd=lm(data_bd2) 
vif(fit_bd) 
summary(fit_bd) 
scale_data_bd2=data.frame(scale(data_bd2))
colnames(scale_data_bd2)=colnames(data_bd2) 
fit_bd_std=lm(scale_data_bd2) 
summary(fit_bd_std) 

Can you please help me understand why sign of regression coefficient differ before and after standardization?

Answer 1

Your code is somehow bugged, and the sign shouldn't change.

The ordinary least squares estimator is: $$ b = (X'X)^{-1} X'y $$

Let $A$ be some invertible linear transformation. Our transformed data is: $$ \hat{X} = X A$$

The OLS estimator on the transformed data is: $$ \begin{align*} \hat{b} &= (A'X'XA)^{-1} A'X'y \\ &= A^{-1} (X'X)^{-1} A'^{-1}A'X'y \\ &= A^{-1} b \end{align*} $$

The coefficients $\hat{b}$ you estimate using the transformed data should be a linear transformation (using $A^{-1}$) of the coefficients $b$ you estimate using the original data.

If you're simply standardizing $X$, what should $A$ look like? If the last column of $X$ is a column of ones (because a constant is including in the regression), then $A$ would be something like: $$ A = \left[ \begin{array}{cccc} \frac{1}{\sigma_x} & 0 & 0 \\ 0 & \frac{1}{\sigma_y} & 0 \\ - \frac{\mu_x}{\sigma_x} & - \frac{\mu_y}{\sigma_y} & 1 \end{array} \right] $$

Multiplying $X$ by $A$ is basically equivalent to subtracting the mean for each non-constant column and dividing by the standard deviation.

$$ A^{-1} = \left[ \begin{array}{cccc} {\sigma_x} & 0 & 0 \\ 0 & {\sigma_y} & 0 \\ \mu_x & \mu_y & 1 \end{array} \right] $$

$$ \hat{b}_x = \sigma_x b_x \quad \quad \hat{b}_y = \sigma_y b_y \quad \quad \hat{b}_1 = b_1 + b_x \mu_x + b_y \mu_y $$

And so the sign of estimates for variables shouldn't change (except the sign for the constant may change). If you don't include a constant though in the regression, all bets are off.