Understanding the deviance and pseudo-R2 from a GLM

Question

I'm investigating some measures of model performance I can use for my (poisson) GLM models and came across a McFadden pseudo R2:

$$ R^2 = 1 - \frac{\text{Residual deviance}}{\text{Null deviance}}, $$

I then went on to read here (p23) that:

this shouldn't be used to compare models which have a different number of parameters on an 'in-sample' dataset because there is no adjustment for the number of degrees of freedom.

Conceptually I think I understand this because deviance always reduces when you add more parameters so models with different number of parameters are not directly comparable. Is that correct?

The author makes the distinction here for the 'in-sample' dataset - does this mean that this measure can be a useful when assessing the hold-out performance of two models with different numbers of parameters? If so, how do we explain this?

Answer 1

You understand correctly the reason for not directly comparing in sample models without taking into account the number of variables. As you add more variables you can overfit and an evaluation method that can account for number of variables is sometimes used, see for example Akaike’s information criterion.

For out-of-sample comparisons, consider the case of adding a spurious regressor to a model containing “real” regressors. Then, when the model is fit that spurious regressor will only add noise when it is evaluated on the out-of-sample data in two ways:

1. Noise from its coefficient.
2. Altering (detrimentally) the signal from the other coefficients.

So whereas for the in-sample case the model will always get better when you add variables, (but you cannot differentiate if this is due to a real effect or not just from McFadden unless it is a large effect), for the out-of-sample use the McFadden value will not increase much due to extra faux variables.

This is a contrived example and fairly informal. In reality a regressor may be poor but still have some use in influencing the dependent variable, and then the “signal” from in and out of sample statistics will be less clear.