1
$\begingroup$

Is it possible to calculate AIC or BIC for nonlinear regression models like SVM, regression trees, artificial neural network, and others. AIC and BIC can be estimated from linear models, but I have not seen AIC and BIC being computed for these nonlinear regression models. So, wondering if anyone can provide their opinion with some examples? Thanks.

$\endgroup$
2
$\begingroup$

The AIC and BIC are both functions of the likelihood. Any model that is fit by maximum likelihood has a straightforward AIC and/or BIC. Some models that are fit with a penalized likelihood can also provide AIC. For example, a generalized additive model provides AIC by counting effective degrees of freedom rather than parameters, and the maximum of the penalized likelihood. I suppose this could be done with ridge regression as well.

One way to think of a regression tree is as a linear regression on dummy variables that indicate which partition the data falls into. If you recast your tree that way, then there is nothing stopping you from fitting it with least squares and then computing AIC the normal way.

But why would you do this when packages like rpart automatically cross-validate for you? Cross-validation is almost always better than AIC for model selection. Other models could probably be shoehorned into a framework that makes AIC computation feasible. But the question will generally be: why?

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Do you know of some work that indeed compared cross-validation with AIC or other information criterions and demonstrated the superior performance of cross-validation with AIC for non-parametric regression models? I ran and compared the regression models using 10 fold cross-validations with five repetitions using CARET packages, and I used R2 and RSME to pick the best models. But an anonymous reviewer asked me to use information criterions to evaluate the effectiveness of models. So, wondering if someone can help me craft an appropriate answer to that question. Your response helps. Thanks. $\endgroup$ – Sami khanal May 23 '18 at 13:45
  • $\begingroup$ Sounds like the reviewer needs to be "handled". The best way to mollify this person probably depends on the precise sort of model you used. $\endgroup$ – generic_user May 23 '18 at 13:57
  • $\begingroup$ I agree, and the only way to do this is to cite some existing works if there are out any without doing any additional computations. Further, I am not a statisticians, and sometimes answering this type of question to stat savvy becomes very tricky. $\endgroup$ – Sami khanal May 23 '18 at 14:55
  • $\begingroup$ You can cite either "Elements of Statistical Learning" or "An Introduction to Statistical Learning" (both free online), but, flipping through, the authors make clear that CV is generally better, though they never explicitly state that they are superior to AIC. I suppose that general statements like that are generalizations, and thus always wrong. $\endgroup$ – generic_user May 23 '18 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.