1
$\begingroup$

I wonder what cause the correlation between parameter estimates in regression analysis?

Does misspecifacation of model cause such a correlation? Or does multicollinearity cause such a correlation?

In the existence of such correlations between parameter estimates, do we need to search the reason and study on the model to decrease such correlations?

I ask this in the sense of simple regression. But in real, I work with a logit model with categorical dependent variables. And in my situation there is strong correlation between dummy varaible's(in the levels of a categorical variable) parameter estimates.

I will be very glad for any help.Thanks a lot.

$\endgroup$
2
$\begingroup$

The source of correlation between parameter estimates is the finite size of the design matrix.

Consider the OLS parameter covariance matrix estimate: $$\operatorname{Var}[\, \hat\beta \mid X \,] = \sigma^2(X ^T X)^{-1}$$ The design matrix columns are usually correlated, that's normal, there's nothing wrong with it at all.

The formula is for finite sample, which is a very important consideration. Why? Because if you collect infinite number of observations, the diagonals of a term $(X'X)$ will become infinite, making uncertainty completely disappear. When there's no uncertainty, the question of the correlations is moot.

Of course, if the design matrix columns are uncorrelated this will also make parameter estimates uncorrelated, but this is truly a rare situation. Almost all design matrices will have variables correlated.

$\endgroup$
0
$\begingroup$

In the simple regression, we assume the error terms are IID. The correlation between parameter estimates comes from collinearity of design matrix. and it is the only source of the correlation. If $X'X$ is diagonal, the parameter estimates are independent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.