13
$\begingroup$

I just ran into "Akaike information criterion", and I noticed this large amount of literature on model selection (also things like BIC seem to exist).

Why don't contemporary machine learning methods take advantage of these BIC and AIC model selection criteria?

$\endgroup$
  • 9
    $\begingroup$ because nobody's calculating the likelihoods? $\endgroup$ – Aksakal Nov 15 '17 at 14:16
  • 1
    $\begingroup$ What do you mean by "contemporary machine learning methods"? As far as I used AIC and BIC are used frequently. $\endgroup$ – Ferdi Nov 15 '17 at 14:16
  • 1
    $\begingroup$ It is simply an adjusted error variance taking into account the sample size and the # of parameters. In my view/religion/belief system it is simply descriptive and not inferential. $\endgroup$ – IrishStat Nov 15 '17 at 14:19
  • 4
    $\begingroup$ Also why the -1? Remember there are no stupid questions -- each question tries to shed light on the universe $\endgroup$ – echo Nov 15 '17 at 14:28
  • 4
    $\begingroup$ @echo: I didn't downvote, but I think your question would be improved if you could source/support the main claim (that machine learning methods do take advantage of these BIC and AIC model selection criteria) $\endgroup$ – user603 Nov 15 '17 at 14:32
9
$\begingroup$

AIC and BIC are used, e.g. in stepwise regression. They are actually part of a larger class of "heuristics", which are also used. For example the DIC (Deviance Information Criterion) is often used in Bayesian Model selection.

However, they are basically "heuristics". While it can be shown, that both the AIC and BIC converge asymptotically towards cross-validation approaches (I think AIC goes towards leave-one-out CV, and BIC towards some other approach, but I am not sure), they are known to under-penalize and over-penalize respectively. I.e. using AIC you will often get a model, which is more complicated than it should be, whereas with BIC you often get a model which is too simplistic.

Since both are related to CV, CV is often a better choice, which does not suffer from these problems.

Then finally there is the issue of the # of parameters which are required for BIC and AIC. With general function approximators (e.g. KNNs) on real-valued inputs, it is possible to "hide" parameters, i.e. to construct a real number which contains the same information as two real numbers (think e.g. of intersecting the digits). In that case, what is the actual number of parameters? On the other hand, with more complicated models, you may have constraints on your parameters, say you can only fit parameters such that $\theta_1 > \theta_2$ (see e.g. here). Or you may have non-identifiability, in which case multiple values of the parameters actually give the same model. In all these case, simply counting of parameters does not give a suitable estimate.

Since many contemporary machine-learning algorithms show these properties (i.e. universal approximation, unclear number of parameters, non-identifiability), AIC and BIC are less useful for these model, than they may seem at first glance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.