1
$\begingroup$

I'm doing a liner regression fit using R. I used lm() to do the regression. Then I standardize my data using scale() and again do the regression on standardize data using lm().

Surprisingly the regression coefficient of one variable was positive before standardization and after standardization I found it is showing negative coefficient. I checked the correlation between that variable and my predictor. It has positive significant correlation.

data_bd2=data_2[,c(1:3,5:7)] 
str(data_bd2) 
fit_bd=lm(data_bd2) 
vif(fit_bd) 
summary(fit_bd) 
scale_data_bd2=data.frame(scale(data_bd2))
colnames(scale_data_bd2)=colnames(data_bd2) 
fit_bd_std=lm(scale_data_bd2) 
summary(fit_bd_std) 

Can you please help me understand why sign of regression coefficient differ before and after standardization?

$\endgroup$
  • $\begingroup$ could you show your code in R? $\endgroup$ – Nick Aug 30 '16 at 7:46
  • $\begingroup$ Hi Nick, here is my code data_bd2=data_2[,c(1:3,5:7)] str(data_bd2) fit_bd=lm(data_bd2) vif(fit_bd) summary(fit_bd) scale_data_bd2=data.frame(scale(data_bd2)) colnames(scale_data_bd2)=colnames(data_bd2) fit_bd_std=lm(scale_data_bd2) summary(fit_bd_std) $\endgroup$ – Python123 Aug 30 '16 at 8:02
  • $\begingroup$ Why don't you use the ''formula='' ? $\endgroup$ – user83346 Aug 30 '16 at 8:13
  • $\begingroup$ Sorry, I didn't ger fcop!! $\endgroup$ – Python123 Aug 30 '16 at 8:16
  • $\begingroup$ Usually you specify a formula in 'lm' like 'lm(formula=y~x, data=scale_data_bd2) ? $\endgroup$ – user83346 Aug 30 '16 at 8:33
4
$\begingroup$

Your code is somehow bugged, and the sign shouldn't change.

The ordinary least squares estimator is: $$ b = (X'X)^{-1} X'y $$

Let $A$ be some invertible linear transformation. Our transformed data is: $$ \hat{X} = X A$$

The OLS estimator on the transformed data is: $$ \begin{align*} \hat{b} &= (A'X'XA)^{-1} A'X'y \\ &= A^{-1} (X'X)^{-1} A'^{-1}A'X'y \\ &= A^{-1} b \end{align*} $$

The coefficients $\hat{b}$ you estimate using the transformed data should be a linear transformation (using $A^{-1}$) of the coefficients $b$ you estimate using the original data.

If you're simply standardizing $X$, what should $A$ look like? If the last column of $X$ is a column of ones (because a constant is including in the regression), then $A$ would be something like: $$ A = \left[ \begin{array}{cccc} \frac{1}{\sigma_x} & 0 & 0 \\ 0 & \frac{1}{\sigma_y} & 0 \\ - \frac{\mu_x}{\sigma_x} & - \frac{\mu_y}{\sigma_y} & 1 \end{array} \right] $$

Multiplying $X$ by $A$ is basically equivalent to subtracting the mean for each non-constant column and dividing by the standard deviation.

$$ A^{-1} = \left[ \begin{array}{cccc} {\sigma_x} & 0 & 0 \\ 0 & {\sigma_y} & 0 \\ \mu_x & \mu_y & 1 \end{array} \right] $$

$$ \hat{b}_x = \sigma_x b_x \quad \quad \hat{b}_y = \sigma_y b_y \quad \quad \hat{b}_1 = b_1 + b_x \mu_x + b_y \mu_y $$

And so the sign of estimates for variables shouldn't change (except the sign for the constant may change). If you don't include a constant though in the regression, all bets are off.

$\endgroup$
  • $\begingroup$ But my sign of estimates for variable is changing not the constant & intuitively also it is looking meaning less to me but I can't able to identify the problem $\endgroup$ – Python123 Aug 30 '16 at 11:12
  • $\begingroup$ They don't only scale X. They scale y too. $\endgroup$ – Roland Aug 30 '16 at 11:40
  • $\begingroup$ To understand the regression they run, check out as.formula(iris[,-5]) $\endgroup$ – Roland Aug 30 '16 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.