22
$\begingroup$

When trying to select among various models or the number of features to include for, say prediction I can think of two approaches.

  1. Split the data into training and test sets. Better still, use bootstrapping or k-fold cross-validation. Train on the training set each time and calculate the error over the test set. Plot test error vs. number of parameters. Usually, you get something like this:enter image description here
  2. Compute the likelihood of the model by integrating over the values of the parameters. i.e., compute $\int_\theta P(D|\theta)P(\theta)d \theta$, and plotting this against the number of parameters. We then get something like this:enter image description here

So my questions are:

  1. Are these approaches suitable for solving this problem (deciding how many parameters to include in your model, or selecting among a number of models)?
  2. Are they equivalent? Probably not. Will they give the same optimal model under certain assumptions or in practice?
  3. Other than the usual philosophical difference of specifying prior knowledge in Bayesian models etc., what are the pros and cons of each approach? Which one would you chose?

Update: I also found the related question on comparing AIC and BIC. It seems that my method 1 is asymptotically equivalent to AIC and method 2 is asymptotically related to BIC. But I also read there that BIC is equivalent to Leave-One-Out CV. That would mean that the training error minimum and Bayesian Likelihood maximum are equivalent where LOO CV is equivalent to K-fold CV. A perhaps very interesting paper "An asymptotic theory for linear model selection" by Jun Shao relates to these issues.

$\endgroup$
  • $\begingroup$ I don't really have a full answer, but I will mention that I would not usually think of using either method to "choose the number of features". In general, I interpret Machine Learning and Bayesian Statistics to just include all the features since they all likely have some level of minimal impact. However, I think the question of relative model complexity is still appropriate. I will also state I've never actually performed the Bayesian inference you allude to; it just seems to get too messy in practice compared to the simplicity of k-fold or bootstrapping. $\endgroup$ – Shea Parkes Jan 8 '12 at 0:17
  • $\begingroup$ Note that Shao paper works only for linear models; in fact only their simple structure makes the number of features usable as a complexity measure and thus power all those information criteria. $\endgroup$ – user88 Jan 9 '12 at 19:28
  • 1
    $\begingroup$ AIC (not BIC!) is asymptotically equivalent to leave-one-out cross validation under weak assumptions (due to Stone "An asymptotic equivalence of choice of model by cross-validation and Akaike's criterion" (1977)). The source in the question you refer to was wrong and was corrected by Rob Hyndman in a comment. I thought it might be a good idea to correct it here, too, to stop spreading the wrong idea. $\endgroup$ – Richard Hardy Feb 25 '15 at 9:59
13
$\begingroup$
  1. Are these approaches suitable for solving this problem (deciding how many parameters to include in your model, or selecting among a number of models)?

Either one could be, yes. If you're interested in obtaining a model that predicts best, out of the list of models you consider, the splitting/cross-validation approach can do that well. If you are interested in known which of the models (in your list of putative models) is actually the one generating your data, then the second approach (evaluating the posterior probability of the models) is what you want.

  1. Are they equivalent? Probably not. Will they give the same optimal model under certain assumptions or in practice?

No, they are not in general equivalent. For example, using AIC (An Information Criterion, by Akaike) to choose the 'best' model corresponds to cross-validation, approximately. Use of BIC (Bayesian Information Criterion) corresponds to using the posterior probabilities, again approximately. These are not the same criterion, so one should expect them to lead to different choices, in general. They can give the same answers - whenever the model that predicts best also happens to be the truth - but in many situations the model that fits best is actually one that overfits, which leads to disagreement between the approaches.

Do they agree in practice? It depends on what your 'practice' involves. Try it both ways and find out.

  1. Other than the usual philosophical difference of specifying prior knowledge in Bayesian models etc., what are the pros and cons of each approach? Which one would you choose?
  • It's typically a lot easier to do the calculations for cross-validation, rather than compute posterior probabilities
  • It's often hard to make a convincing case that the 'true' model is among the list from which you are choosing. This is a problem for use of posterior probabilities, but not cross-validation
  • Both methods tend to involve use of fairly arbitrary constants; how much is an extra unit of prediction worth, in terms of numbers of variables? How much do we believe each of the models, a priori?
    • I'd probably choose cross-validation. But before committing, I'd want to know a lot about why this model-selection was being done, i.e. what the chosen model was to be used for. Neither form of model-selection may be appropriate, if e.g. causal inference is required.
$\endgroup$
16
$\begingroup$

Optimisation is the root of all evil in statistics! ;o)

Anytime you try to select a model based on a criterion that is evaluated on a finite sample of data, you introduce a risk of over-fitting the model selection criterion and end up with a worse model than you started with. Both cross-validation and marginal likelihood are sensible model selection criteria, but they are both dependent on a finite sample of data (as are AIC and BIC - the complexity penalty can help, but doesn't solve this problem). I have found this to be a substantial issue in machine learning, see

G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, 2010. Research, vol. 11, pp. 2079-2107, July 2010. (www)

From a Bayesian point of view, it is better to integrate over all model choices and parameters. If you don't optimise or choose anything then it becomes harder to over-fit. The downside is you end up with difficult integrals, which often need to be solved with MCMC. If you want best predictive performance, then I would suggest a fully Bayesian approach; if you want to understand the data then choosing a best model is often helpful. However, if you resample the data and end up with a different model each time, it means the fitting procedure is unstable and none of the models are reliable for understanding the data.

Note that one important difference between cross-validation and evidence is that the value of the marginal likelihood assumes that the model is not misspecified (essentially the basic form of the model is appropriate) and can give misleading results if it is. Cross-validation makes no such assumption, which means it can be a little more robust.

$\endgroup$
  • $\begingroup$ Bayesian integration is a strong approach. But always question whether model selection is even the right way to go about this. What is the motivation? Why not posit a complete model that is flexible and just fit it? $\endgroup$ – Frank Harrell Jan 30 '18 at 12:34
  • $\begingroup$ @FrankHarrell many flexible models include regularisation terms and other hyper-parameters, and tuning those is also model selection and subject to the same problems of over-fitting the selection criterion. Fitting introduces a risk of over-fitting, and that applies at all levels. However if you know a-priori about the structure of the model, then that expert knowledge should be used. $\endgroup$ – Dikran Marsupial Feb 1 '18 at 7:23
  • 1
    $\begingroup$ Best to seek a method that doesn't require tuning, but this isn't always possible. My main point is that model specification works better than model selection and don't assume that feature selection is a noble goal. $\endgroup$ – Frank Harrell Feb 1 '18 at 11:53
  • $\begingroup$ @FrankHarrell feature selection is very rarely helpful. Optimisation should be avoided where possible, which involves making any model choice/tuning based on a finite sample of data (of course the larger the sample, the lower the risk). $\endgroup$ – Dikran Marsupial Feb 2 '18 at 8:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.