# Comparing models with different number of predictors

Given that the overall F-test of a multiple regression model has an F distribution, which depends on the number of predictors in the model, I understand why you cannot compare the F-statistics from models with different number of predictors.

However, the p-value of the F-statistics always has a uniform distribution between 0 and 1, and represents the probability that at least one βj ≠ 0 under the null hypothesis.

Can I compare the F-statistics p-value from models with different number of predictors? If not, what are good alternatives? (R-squared?; adjusted R-squared?)

• I guess I should have emphasized that I'm more interested in whether comparing the F-statistics p-value is a valid approach. – Marcelo Mattar Nov 12 '13 at 17:48

I would NOT recommend the $R^2$ as this measure increases as the number of variables increases. In other words, the $R^2$ does not account for overfitting.

Among the options you mentioned the adjusted $R^2$ would be the best. If you take a look at the formula:

$R^2_{adj} = 1 - \frac{(1-R^2)\cdot(n-1)}{n-p-1}$

Since the number of parameters $p$ is in the denominator of the formula, the addition of variables that do not increase significantly the $R^2$ will penalize the $R^2_{adj}$.

A better approach to compare your models would be to use the Akaike Information Criterion:

$AIC_i = -2\cdot log(\mathcal{L}_i) + 2\cdot p_i$

where $\mathcal{L}_i$ is the likelihood of model $i$

You could obtain this very easy in R by using the AIC function:  AIC(model1, model2)

• Thanks a lot for your answer, Orlando. However, would you be able to explain why I shouldn't compare the F-statistic p-value? – Marcelo Mattar Nov 7 '13 at 1:01