I am running a regression analysis to understand the effect of several IVs on the transport mode choice of questionnaire respondents. My sample of respondents is of 100, and I have more than 10 categorical variables I would like to examine as IVs. If I use all of them together on a model, I do not find anyone to be significant, due to my sample size. Thus, my question is: does it make sense to test sets of these variables that are theoretically related in separate models, and then combine in a model only those that were significant?

  • 2
    $\begingroup$ If only one of them could be important, it would make sense to do univariate tests (though then you might worry about overall type I error rate). If several variables could be important, you may not get any of the univariate results show anything; and even if you do manage to identify some, the signs of the coefficients might be wrong (compared to what they would be in a model that had all the important predictors in it). $\endgroup$ – Glen_b Jan 12 '15 at 0:46
  • 1
    $\begingroup$ Consider PCA or some other ways of combining IVs before using them in the regression. $\endgroup$ – Aksakal Jan 12 '15 at 15:10
  • $\begingroup$ What size correlations are you seeing between your IVs and the DV (transport mode)? Are these in the direction and of the size you expected? If not, can you find a reason for the discrepancy? If there is no previous research in this area, it may well be that the IVs you are looking at are the wrong IVs or the measures you have are too crude. $\endgroup$ – Joel W. Jan 14 '15 at 18:23

I would be wary of that approach. If all of the IVs are orthogonal to one another (that is, they are independent of one another), then there'd be no difference between 10 univariate tests and 1 multiple regression test, except fewer residual df. However, in practice, I'd bet that among 10 IVs, at least 2 or 3 of them are correlated with one another.

Multiple regression tells you what the influence of one IV is holding all others constant. If IVs are correlated, then this yields better inference about the particular effects of each IV. The first thing I would do is check for collinearity. Maybe some IVs are highly correlated with one another and therefore redundant. You can then exclude redundant IVs. Another thing I would try would be some kind of factor analysis on the IVs, usually PCA. You might be able to use PCA on the IVs to reduce the 10 IVs into 3-4 IVs, depending on how correlated the IVs are. This would take advantage of your full set of IVs and allow you to use multiple regression at the same time.

Another approach would be to formulate specific a-priori hypotheses given particular combinations of IVs. You can then use model selection (AIC, AICc) to identify which sets of IVs are important and then examine parameter estimates and SEs from there. See Burnham and Anderson (2002) - Model selection and multimodel inference for more details.

Either way, you should find an approach that allows you to carefully make use of all IVs, rather than some kind of stepwise approach. There are too many issues with data dredging and multiple comparisons. You might be missing something in the univariate tests (such as one variable unimportant on its own, but important in the presence of others).

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.