Model parameters, statistical significance and errors

Question

I am fitting a basic generalized linear model (binomial) to some survey data that has a small sample size (N=376). The model my colleague has fit has 23 model parameters. Two or three have p-values less than 0.1.
There are theoretical reasons to expect that those particular independent variables are linked to the dependent variable, so that's good. But, with so many independent variables included in a model, particularly with a reasonably small sample size, is it not the case that the risk of a type I error increases? In other words, putting so many model parameters into the model, are we not increasing the chances that some parameters will present themselves as statistically significant, just out of random chance?

To test this, would it be generally adviseable to begin removing non-statistically significant independent variables from the model to see if the remaining parameters remain significant?

I think I'm sensitive to the danger of chasing the stars of statistical significance, but I want to be sure my logic and modelling strategy is correct.

S.

Answer 1

Generally, model building through stepwise regression is a good way to insure biased results. See for example:

Steyerberg, E. W., Eijkemans, M. J., and Habbema, J. D. F. (1999). Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. Journal of clinical epidemiology, 52(10):935–942.

Whittingham, M., Stephens, P., Bradbury, R., and Freckleton, R. (2006). Why do we still use stepwise modelling in ecology and behaviour? Journal of Animal Ecology, 75(5):1182–1189.

If concerned about inflated Type I error from testing so many hypotheses/including so many predictors, consider methods for controlling the family-wise error rate or the false discovery rate.

Answer 2

If you have 23 parameters and 2 or 3 have p value under 0.1 then.... well that's about what you would expect if there were no relationship in the whole population. I would view these results with suspicion.

It is good that you have theoretical reasons to include the variables; do those reasons include estimates of effect size? How do the effect sizes you got compare with the ones you expect? I would guess they are smaller, which may say something about your sample compared to the ones that the theory is based on.

Answer 3

You could check for correlation between the independent variables. Since there are too many of them, they can interact within themselves, thus affecting the significance level of the others. start by removing the variables that are highly correlated.