# Why does the sign of Regression coefficients for standardized & unstandardize data using R differ

I'm doing a liner regression fit using R. I used lm() to do the regression. Then I standardize my data using scale() and again do the regression on standardize data using lm().

Surprisingly the regression coefficient of one variable was positive before standardization and after standardization I found it is showing negative coefficient. I checked the correlation between that variable and my predictor. It has positive significant correlation.

data_bd2=data_2[,c(1:3,5:7)]
str(data_bd2)
fit_bd=lm(data_bd2)
vif(fit_bd)
summary(fit_bd)
scale_data_bd2=data.frame(scale(data_bd2))
colnames(scale_data_bd2)=colnames(data_bd2)
fit_bd_std=lm(scale_data_bd2)
summary(fit_bd_std)

• could you show your code in R? – Nick Aug 30 '16 at 7:46
• Hi Nick, here is my code data_bd2=data_2[,c(1:3,5:7)] str(data_bd2) fit_bd=lm(data_bd2) vif(fit_bd) summary(fit_bd) scale_data_bd2=data.frame(scale(data_bd2)) colnames(scale_data_bd2)=colnames(data_bd2) fit_bd_std=lm(scale_data_bd2) summary(fit_bd_std) – Python123 Aug 30 '16 at 8:02
• Why don't you use the ''formula='' ? – user83346 Aug 30 '16 at 8:13
• Sorry, I didn't ger fcop!! – Python123 Aug 30 '16 at 8:16
• Usually you specify a formula in 'lm' like 'lm(formula=y~x, data=scale_data_bd2) ? – user83346 Aug 30 '16 at 8:33

Your code is somehow bugged, and the sign shouldn't change.

The ordinary least squares estimator is: $$b = (X'X)^{-1} X'y$$

Let $A$ be some invertible linear transformation. Our transformed data is: $$\hat{X} = X A$$

The OLS estimator on the transformed data is: \begin{align*} \hat{b} &= (A'X'XA)^{-1} A'X'y \\ &= A^{-1} (X'X)^{-1} A'^{-1}A'X'y \\ &= A^{-1} b \end{align*}

The coefficients $\hat{b}$ you estimate using the transformed data should be a linear transformation (using $A^{-1}$) of the coefficients $b$ you estimate using the original data.

If you're simply standardizing $X$, what should $A$ look like? If the last column of $X$ is a column of ones (because a constant is including in the regression), then $A$ would be something like: $$A = \left[ \begin{array}{cccc} \frac{1}{\sigma_x} & 0 & 0 \\ 0 & \frac{1}{\sigma_y} & 0 \\ - \frac{\mu_x}{\sigma_x} & - \frac{\mu_y}{\sigma_y} & 1 \end{array} \right]$$

Multiplying $X$ by $A$ is basically equivalent to subtracting the mean for each non-constant column and dividing by the standard deviation.

$$A^{-1} = \left[ \begin{array}{cccc} {\sigma_x} & 0 & 0 \\ 0 & {\sigma_y} & 0 \\ \mu_x & \mu_y & 1 \end{array} \right]$$

$$\hat{b}_x = \sigma_x b_x \quad \quad \hat{b}_y = \sigma_y b_y \quad \quad \hat{b}_1 = b_1 + b_x \mu_x + b_y \mu_y$$

And so the sign of estimates for variables shouldn't change (except the sign for the constant may change). If you don't include a constant though in the regression, all bets are off.

• But my sign of estimates for variable is changing not the constant & intuitively also it is looking meaning less to me but I can't able to identify the problem – Python123 Aug 30 '16 at 11:12
• They don't only scale X. They scale y too. – Roland Aug 30 '16 at 11:40
• To understand the regression they run, check out as.formula(iris[,-5]) – Roland Aug 30 '16 at 11:44