When we were doing the linear regression, let's say the reduced model $y=\beta_0+\beta_1x_1+\beta_2x_2+\epsilon$ is the true model (i.e. we were using this model to generate data), then how can we tell this model is better than the full model say $y=\beta_0+\beta_1x_1+\beta_2x_2+\beta_3x_3+\epsilon$? Since the regression results could give $\beta_2$ to be some very small number, and the BIC and AIC between these two models are quite similar.
I will give some of my numerical results as:
#below using mu=rep(0,4), D is a 4x4 matrix with D[4,4]=0, so we force beta=0 beta <- mvrnorm(n = 1, mu, D, tol = 1e-6, empirical = FALSE, EISPACK = FALSE) epsilon <- mvrnorm(n = 1, rep(0,8), diag(8), tol = 1e-6, empirical = FALSE, EISPACK = FALSE) y <- X%*%beta+epsilon # the design matrix X is a 8x4 matrix defined previously
The BIC value actually varies depends on the data (I mean the second model not always has the smallest BIC), but the first two model is always obvious better than the last two:
> BIC(lm(y ~ X))  -520.8597 > BIC(lm(y ~ X[,1:3]))  -522.8282 > BIC(lm(y ~ X[,1:2]))  27.37185 > BIC(lm(y ~ X[,1]))  53.32879
And when I print out the
lm results on
lm(y ~ X), the last coefficient is very small, like very close to zero. I guess that is why we can not distinguish the first two models very well.
Since when the reduced model is true then the full model is also true. So is there any way to compare the full model with the reduced models?