1
$\begingroup$

I heard that multiple regression assumes that the independent variables are

correlated somehow. So when we convert the multiple regression into SEM diagram,

we see covariance arrows are drawn between independent variables.

So I wonder what if we assume the independecy between independent variables?

Does it change the estimates of coefficient? How much would it make difference?

And in what conditions would the difference get larger?

If possible, some mathematical background would be appreciated.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ The nice stuff about multiple regression is that the regressors may be correlated. $\endgroup$
    – Michael M
    Mar 30, 2015 at 15:50

1 Answer 1

Reset to default
4
$\begingroup$

The model for basic linear regression is $Y = X\beta + \varepsilon$ where $\varepsilon \sim N(0, \sigma^2 I_n)$. We model $E(Y|X)$. The $X$'s are considered fixed and without measurement error. There are 3 assumptions being made here. They are

  1. a linear model is correct
  2. the errors are normal with mean 0 and constant variance
  3. the errors are all independent.

Note that as a result of this $E(Y|X) = X\beta$ and $Var(Y|X) = \sigma^2I_n$. We also have that $\forall i \neq j \ \ Y_i \perp Y_j$. We could also formulate this by having $Var(\varepsilon) = W$ where $W$ is diagonal or $\Sigma$ where $\Sigma$ is some symmetric and positive definite matrix. These would each lead to different estimates of $\beta$ because $Var(\varepsilon)$ is a component of the solution to $\hat{\beta}$.

Absolutely no assumptions are made about $X$ other than that it is fixed and without measurement error. We may additionally want to assume that it is full rank and that $p < n$ but that's purely because of obtaining $\hat{\beta}$, not because of the model itself. The columns $X_i$ of $X$ can be highly correlated and might give us trouble with $(X^TX)^{-1}$ but that does not violate the model.

Does this address your question?

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for the wonderful answer! Anyway, I heard Multiple Regression is equivalent to path analysis with correlated predictors. Is this correct? If I want to do Multiple Regression with any of path analysis tool, how should I draw the diagram? $\endgroup$
    – yourock
    Mar 30, 2015 at 19:56
  • $\begingroup$ I am getting it. Path analysis implicitly assumes the independence between any 2 variables (or residuals) which is not connected by an arrow, and that's why multiple regression is translated into path diagram with arrows between predictors. Given that, assuming independence between predictors is a nested model of no assumption between predictors. Then does assuming independence between predictors make a large difference? when, and how? $\endgroup$
    – yourock
    Mar 31, 2015 at 0:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.