I would like to determine which of a set of candidate predictors $\{x_1, x_2,\ldots, x_n\}$ are significantly relevant to the linear prediction of $y$.

Typically, one can compare a full model

$$\hat y = a_0 + a_1x_1 + \cdots + a_nx_n$$

to a reduced model

$$\hat y = a_0 + a_1x_1 + \cdots + a_{i-1}x_{i-1} + a_{i+1}x_{i+1} + \cdots + a_nx_n$$

(where the reduced model does not include $x_i$ as one of the predictors) using an F-test, which is based on the sum of the squared residuals for each model. Then, if the p-value is below a certain threshold one can say that $x_i$ is significant.

In my case, however, there are definitely correlations among the predictors, and there may be unknown dependencies among them.

Will this affect the interpretation of the F-test p-value, and if so, are there anyways to account for these dependencies in calculating predictor significance?

This doesn't seem like that uncommon of an obstacle in statistics and may well have been answered before, so directions to relevant resources would also be very much appreciated.

Thanks in advance!

  • 2
    $\begingroup$ Some amount of correlation among predictors is normal in multiple regression. How strong are they in your case? Consider computing variance inflation factors or the condition number for your set of predictors. If excessive, consider Wikipedia's "Remedies for multicollinearity". $\endgroup$ – Nick Stauner Jul 21 '14 at 23:31
  • $\begingroup$ Thanks for the leads. The correlations are fairly minor, and I believe the reduced model is yielding a rather different prediction than the full model, but I would like a more formal way to prove it. I will look up variance inflation and condition number. $\endgroup$ – rkp Jul 22 '14 at 19:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.