I have two statistical models. Model 1 uses a GLM approach while model 2 uses a time series approach for fitting. I want to compare these two models.

Model 1 (i.e. GLM) has a better out of sample performance. Model 2 has a better BIC criteria. So based on out of sample performance, I should pick up model 1 and based on BIC I should pick up model 2 as the preferred model.

I should add that in this context and for the question I am trying to answer, Both the BIC and out of sample performance are important. The question is how to choose the best model in this case? Should I consider other criteria? Please let me know if you know any good reference with similar cases.

  • $\begingroup$ (1) I'm not sure you can necessarily compare a GLM and a time series model via BIC. (2) In any case which you used depends on what you want to do well at; even when BIC's are comparable, BIC is no guarantee of out of sample performance. Why do you want to optimize on one or the other? $\endgroup$
    – Glen_b
    Commented Oct 12, 2013 at 0:09
  • $\begingroup$ Do you have any reference showing that we cannot compare GLM and time series using BIC? Because to me, it is possible since BIC just depends on estimated log likelihood and number of parameter and number of observations. These models can be used to price some products and you want your price to be unique. So at the end you need to pick up one. $\endgroup$
    – Stat
    Commented Oct 12, 2013 at 0:24
  • 1
    $\begingroup$ Having seen the particular assumptions under which BIC was derived, I don't see how the comparisons implied by that derivation applies to your situation; the onus would be yours to show that what you're doing makes sense. [In fact I have one reference that says you can't compare likelihoods across models with different error distributions, which if it were correct would wipe out a lot more than just BIC. I don't know that the claim of the reference is correct, though.] $\endgroup$
    – Glen_b
    Commented Oct 12, 2013 at 1:02
  • 2
    $\begingroup$ @Glen_b I believe that this paper of Vuong (1989), jstor.org/discover/10.2307/1912557 provides a general framework for non-nested models. $\endgroup$ Commented Oct 12, 2013 at 14:26
  • 4
    $\begingroup$ Possible duplicate of Can AIC compare across different types of model? $\endgroup$ Commented Aug 31, 2018 at 16:06


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.