Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am fitting a basic generalized linear model (binomial) to some survey data that has a small sample size (N=376). The model my colleague has fit has 23 model parameters. Two or three have p-values less than 0.1.
There are theoretical reasons to expect that those particular independent variables are linked to the dependent variable, so that's good. But, with so many independent variables included in a model, particularly with a reasonably small sample size, is it not the case that the risk of a type I error increases? In other words, putting so many model parameters into the model, are we not increasing the chances that some parameters will present themselves as statistically significant, just out of random chance?

To test this, would it be generally adviseable to begin removing non-statistically significant independent variables from the model to see if the remaining parameters remain significant?

I think I'm sensitive to the danger of chasing the stars of statistical significance, but I want to be sure my logic and modelling strategy is correct.

S.

share|improve this question
add comment

2 Answers 2

Generally, model building through stepwise regression is a good way to insure biased results. See for example:

Steyerberg, E. W., Eijkemans, M. J., and Habbema, J. D. F. (1999). Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. Journal of clinical epidemiology, 52(10):935–942.

Whittingham, M., Stephens, P., Bradbury, R., and Freckleton, R. (2006). Why do we still use stepwise modelling in ecology and behaviour? Journal of Animal Ecology, 75(5):1182–1189.

If concerned about inflated Type I error from testing so many hypotheses/including so many predictors, consider methods for controlling the family-wise error rate or the false discovery rate.

share|improve this answer
add comment

If you have 23 parameters and 2 or 3 have p value under 0.1 then.... well that's about what you would expect if there were no relationship in the whole population. I would view these results with suspicion.

It is good that you have theoretical reasons to include the variables; do those reasons include estimates of effect size? How do the effect sizes you got compare with the ones you expect? I would guess they are smaller, which may say something about your sample compared to the ones that the theory is based on.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.