3
$\begingroup$

I don't understand very well the statement that in the face of multi-collinearity we may fail to reject the null hypothesis.

The Null Hypothesis is that the coefficient of a variable is zero. This means that it does not contribute really to the model. Now let a variable X1 be an important predictor of a variable Y and X2 another variable closely correlated with X2. In the presence of X1 , X2 is clearly not needed. This may not be apparent in a model containing both X1 and X2 as both will receive credit for the influence of X1 on Y , although the p value of X2 many times is high enough to consider it insignificant. In case X1 is absent, X2 will receive all the credit, although it is not the true driver of Y but rather an indicator of the latent influence of X1.

Given the above could you help me understand the meaning of the phrase: Multi-collinearity may cause failure to reject the Null Hypothesis?

Your advice will be appreciated.

$\endgroup$

2 Answers 2

2
$\begingroup$

With multicollinearity there's a set of combinations of values of parameters that correspond to almost the same fit. In the simplest case you could have two highly correlated parameters*. Then your fit criterion ("loss function", e.g. sum of squares of residuals$^\dagger$) would have a long valley in it:

SSE for model with two highly correlated parameters

where there are many combinations of the parameters that have almost the same fit. This means that for any one of the involved parameters (in that multicollinear relationship), you really can't be sure about its value -- a value quite a distance away will be almost as good, as the other parameter(s) involved can compensate to give almost the same fit. Consequently the parameters in the multicollinear relationship will have a large standard error.

Note that in the diagram both values for $\beta_2$ that are close to $0$ and values quite far from $0$ are similar in fit. As a result we really can't tell for sure the population parameter isn't zero. If you didn't have that high degree of collinearity, you might well be able to distinguish that $\beta_2$ was not consistent with $0$.

You can determine the standard error of a particular combination of the parameters fairly accurately (for standardized IVs with positive correlation it would be close to their sum -- in the picture above imagine slicing across that valley you can find the minimum easily), but the other direction (say their difference) you can get almost no information about (slicing along the bottom of the valley).

* though multicollinearity doesn't require any pair of parameters to be highly correlated, if enough of them are involved.

$\dagger$ more generally you could substitute $-\log \mathcal L$ or pretty much any other suitable criterion

$\endgroup$
2
$\begingroup$

The math answer can be provided, but I think I will try to keep this mostly to words. Simply put, multi-collinearity inflates the variances of the coefficient estimates. Consider your estimated equation,

$$ \hat y = x_1\hat\beta_1+x_2\hat\beta_2 $$

The regression model (OLS) will return a parameter estimate, denoted by $ \hat\beta_i $, and standard error, denoted by $ se[\hat\beta_i] $, for each of the estimated parameters, $\hat\beta_1$ and $\hat\beta_2$. The t-statistic is the ratio:

$$ t_{\hat\beta_i} = \frac{\hat\beta_i} {se[\hat\beta_i]} $$

High multi-collinearity (i.e., strong correlation between $X1$ and $X2$) will cause these standard errors to be artificially high, or inflated. Therefore the t-statistics will be artificially low, resulting in high p-values. With "strong" multi-collinearity, these standard errors can be much greater than the mean. This returns t-statistics less than 1, and in turn a p-value well above a traditional $ \alpha=0.05$ cutoff. Therefore, a predictor that is actually in the true model will be overlooked, since we will fail to reject the null hypothesis that it is equal to zero (i.e., not significant). Thus your phrase, "Multi-collinearity may cause failure to reject the Null Hypothesis".

$\endgroup$
4
  • $\begingroup$ Be careful on the math writing. What does $E$ mean in your writing? $\endgroup$
    – user158565
    Jun 12, 2017 at 0:00
  • $\begingroup$ Good point on clarity. Does this make it more clear? $\endgroup$
    – Adam
    Jun 12, 2017 at 12:31
  • $\begingroup$ If you follow the convenient writing in statistics, i.e., ^ means estimate, $E$ means expectation, then OLS does not return $E(\hat\beta)$, and $t$ statistics that you gave is wrong. $\endgroup$
    – user158565
    Jun 12, 2017 at 13:20
  • $\begingroup$ Yeah, you are correct. Had my mind somewhere else. I think I have this right now. Thank you. $\endgroup$
    – Adam
    Jun 12, 2017 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.