perform BIC to compare the full model and the reduced model

Question

When we were doing the linear regression, let's say the reduced model $y=\beta_0+\beta_1x_1+\beta_2x_2+\epsilon$ is the true model (i.e. we were using this model to generate data), then how can we tell this model is better than the full model say $y=\beta_0+\beta_1x_1+\beta_2x_2+\beta_3x_3+\epsilon$? Since the regression results could give $\beta_2$ to be some very small number, and the BIC and AIC between these two models are quite similar.

I will give some of my numerical results as:

#below using mu=rep(0,4), D is a 4x4 matrix with D[4,4]=0, so we force beta[4]=0
beta <- mvrnorm(n = 1, mu, D, tol = 1e-6, empirical = FALSE, EISPACK = FALSE) 
epsilon <- mvrnorm(n = 1, rep(0,8), diag(8), tol = 1e-6, empirical = FALSE, EISPACK = FALSE)
y <- X%*%beta+epsilon # the design matrix X is a 8x4 matrix defined previously

The BIC value actually varies depends on the data (I mean the second model not always has the smallest BIC), but the first two model is always obvious better than the last two:

> BIC(lm(y ~ X))
[1] -520.8597
> BIC(lm(y ~ X[,1:3]))
[1] -522.8282
> BIC(lm(y ~ X[,1:2]))
[1] 27.37185 
> BIC(lm(y ~ X[,1]))
[1] 53.32879

And when I print out the lm results on lm(y ~ X), the last coefficient is very small, like very close to zero. I guess that is why we can not distinguish the first two models very well.

Since when the reduced model is true then the full model is also true. So is there any way to compare the full model with the reduced models?

Answer 1

So the way I have always approached these problems is to employ parsimony (although there is a debate as to whether that is a fair rule to follow) - the simplest model that adequately describes the data is the one that you go with. In the example above, the model:

 BIC(lm(y ~ X[,1:3]))

appears to be the one with the lowest BIC, although if the difference between this one and the full model is significant is another matter - this could be compared with a likelihood ratio test (LRT) or an ANOVA which does approximately the same thing. If the two are not significantly different from one another, then most people assume parsimony and go with the reduced model.

It is important to remember in this regard that it is the best fitting model of the data in question - not necessarily the model which is "true".