# Efficiency and the number of regressors?

An econometrician told me that I shouldn't keep adding new variables to the model even if I have reason to believe they're relevant to the response variable, as it "reduces the efficiency of the other parameters". That is, even if you have 40 variables that "should" be related to the response, you should draw the line somewhere and include a subset of them.

Question: What are some good references that deal with this?

• A nice reference on this is The Cost of Adding Parameters to a Model. On the other hand, a reduction in the efficiency of the estimation is not a criterion for selecting variables/models. – user10525 Nov 3 '12 at 17:51

Probably the best regarded book on the model building process per se, is Frank Harrell's Regression Modeling Strategies, but the issues involved can be stated simply: For every additional covariate that is included in a model, you will lose 1 degree of freedom. If a factor with $k$ levels is included, you will lose $k-1$ degrees of freedom. This will decrease your statistical power (the ability to differentiate the slope of the relationship between that covariate and the response from 0). Another way of putting that fact is that your confidence intervals around your beta estimate will be wider / sample parameters will vary more widely from their true values. If the covariates are orthogonal to each other and you have enough data, the impact is likely to be very small. Real-world (observational rather than experimental) data is never orthogonal, though, so there will be multicollinearity, and multicollinearity can cause your beta estimates to fluctuate quite widely. (There are many threads on CV that explore multicollinearity, so if you aren't terribly familiar with it, you can read some of them by clicking on .)

• I would change a few things about this answer. First, I wouldn't mention df so prominently, since that's only an issue of power, and reducing df by a few becomes irrelevant at the N most people would be dealing with when testing 40-ish predictors. But also, the effects described here on parameter estimates are not simply "another way of putting" the power problem (though probably you didn't mean to imply that). Multicollinearity is a separate problem. And it matters not just for the most recently added predictor(s) but, as the OP's econometrician indicates, all predictors. – rolando2 Nov 3 '12 at 21:17

The proof of this is a standard exercise in matrix algebra, and you can find it in all the graduate econometrics textbooks.

For example, Davidson and MacKinnon (2003) has it on page 112.

I would look into using the Aikake Information Criterion, or the Bayesian Information Criterion.

What software are you using? Most statistical software should take care of this automatically for you.

Essentially they are ways of determining if the number of regression parameters is justified in terms of how much explanatory value they add to the model.

• I'm more looking for a thorough description of what's exactly occurring to efficiency when you add a large number of predictor variables that each have a strong bivariate relationship with the response. – user14281 Nov 3 '12 at 15:29
• Ok, hard to go into what the AIC and BIC mean without knowing more about you. Do you know much about maximum likelihood estimators? – Simon Hayward Nov 3 '12 at 15:30
• No, I don't know much about maximum likelihood estimators. But please feel free to answer with the assumption that I do know everything about it, as most people reading this question in the future will probably have a working knowledge of this. – user14281 Nov 3 '12 at 16:44