24
$\begingroup$

Given a hierarchical model $p(x|\phi,\theta)$, I want a two stage process to fit the model. First, fix a handful of hyperparameters $\theta$, and then do Bayesian inference on the rest of the parameters $\phi$. For fixing the hyperparameters I am considering two options.

  1. Use Empirical Bayes (EB) and maximize the marginal likelihood $p(\mbox{all data}|\theta)$ (integrating out the rest of the model which contains high dimensional parameters).
  2. Use Cross Validation (CV) techniques such as $k$-fold cross validation to choose $\theta$ that maximizes the likelihood $p(\mbox{test data}|\mbox{training data}, \theta)$.

The advantage of EB is that I can use all data at once, while for CV I need to (potentially) compute the model likelihood multiple times and search for $\theta$. The performance of EB and CV are comparable in many cases (*), and often EB is faster to estimate.

Question: Is there a theoretical foundation that links the two (say, EB and CV are the same in the limit of large data)? Or links EB to some generalizability criterion such as empirical risk? Can someone point to a good reference material?

(*) As an illustration, here is a figure from Murphy's Machine Learning, Section 7.6.4, where he says that for ridge regression both procedures yield very similar result:

murphy - empirical bayes vs CV

Murphy also says that the principle practical advantage of the empirical Bayes (he calls it "evidence procedure") over CV is when $\theta$ consists of many hyper-parameters (e.g. separate penalty for each feature, like in automatic relevancy determination or ARD). There it is not possible to use CV at all.

Improve this question
$\endgroup$
4
  • $\begingroup$ Can you describe in more detail what you're doing for the cross-validation method? Are you fixing $\theta$ and then using the training data to estimate the other parameters before validating? $\endgroup$
    – Neil G
    Commented Mar 17, 2012 at 22:55
  • $\begingroup$ @NeilG maximizing the sum of log marginal predictive data likelihood on cross-validation sets (k is integrated out). $\endgroup$
    – Memming
    Commented Mar 17, 2012 at 23:32
  • 1
    $\begingroup$ If $k$ is integrated out both times, then what's the difference between CV and EB? $\endgroup$
    – Neil G
    Commented Mar 18, 2012 at 0:30
  • 2
    $\begingroup$ Great question. I took the liberty to add a figure from Murphy's textbook to your question to illustrate your point about two procedures often being comparable. I hope you will not mind this addition. $\endgroup$
    – amoeba
    Commented Jan 27, 2017 at 10:48

3 Answers 3

Reset to default
18
$\begingroup$

I doubt there will be a theoretical link that says that CV and evidence maximisation are asymptotically equivalent as the evidence tells us the probability of the data given the assumptions of the model. Thus if the model is mis-specified, then the evidence may be unreliable. Cross-validation on the other hand gives an estimate of the probability of the data, whether the modelling assumptions are correct or not. This means that the evidence may be a better guide if the modelling assumptions are correct using less data, but cross-validation will be robust against model mis-specification. CV is assymptotically unbiased, but I would assume that the evidence isn't unless the model assumptions happen to be exactly correct.

This is essentially my intuition/experience; I would also be interested to hear about research on this.

Note that for many models (e.g. ridge regression, Gaussian processes, kernel ridge regression/LS-SVM etc) leave-one-out cross-validation can be performed at least as efficiently as estimating the evidence, so there isn't necessarily a computational advantage there.

Addendum: Both the marginal likelihood and cross-validation performance estimates are evaluated over a finite sample of data, and hence there is always a possibility of over-fitting if a model is tuned by optimising either criterion. For small samples, the difference in the variance of the two criteria may decide which works best. See my paper

Gavin C. Cawley, Nicola L. C. Talbot, "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation", Journal of Machine Learning Research, 11(Jul):2079−2107, 2010. (pdf)

Improve this answer
$\endgroup$
10
  • $\begingroup$ Why do you say that CV is robust against a mis-specified model? In his case, there is no such protection since cross-validation is searching over the same space that EB is calculating a likelihood. If his modelling assumptions are wrong, then cross-validation won't save him. $\endgroup$
    – Neil G
    Commented Mar 18, 2012 at 0:40
  • 2
    $\begingroup$ CV is robust against misspecification in the sense that it still gives a useful indicator of generalisation performance. The marginal likelihood may not as it depends on the prior on $\phi$ (for example), even after you have marginalised over $\phi$. So if your prior on $\theta$ was misleading, the marginal likelihood may be a misleading guide to generalisation performance. See Grace Wahba's monograph on "spline models for observational data", section 4.8 (it doesn't say a great deal, but there isn't much on this topic AFAIK). $\endgroup$
    – Dikran Marsupial
    Commented Mar 19, 2012 at 9:44
  • $\begingroup$ p.s. I have been performing an analysis of avoiding overfitting in neural networks with Bayesian regularisation where the regularisation parameters are tuned via marginal likelihood maximisation. There are situations where this works very badly (worse than not having any regularisation at all). This seems to be a problem of model mis-specification. $\endgroup$
    – Dikran Marsupial
    Commented Mar 19, 2012 at 9:45
  • $\begingroup$ He can get the same "indicator of generalization performance" by checking the total log-probability of the data given the estimated distribution returned by EB (which will be equal to the entropy of that distribution). There's no way to beat it in this case because it is the analytical solution to this problem. I don't see why cross-validation would makes sense when you can calculate a likelihood for EB. $\endgroup$
    – Neil G
    Commented Mar 19, 2012 at 9:50
  • 2
    $\begingroup$ @probabilityislogic, I'm not quite sure what you are getting at (problem undoubtely at my end! ;o). I can tell you from practical experience though that the issue is very real. I have been working on problems in model selection for several years, and I have come across many problems where maximising the marginal likelihood turns out to be a very bad idea. Cross-validation performs about as well for most datasets, but where it performs badly it rarely performs catastrophically as evidence maximisation sometimes does. $\endgroup$
    – Dikran Marsupial
    Commented Mar 19, 2012 at 11:12
0
$\begingroup$

There is actually a paper that connects CV and EB:

E Fong, C C Holmes, On the marginal likelihood and cross-validation, Biometrika, Volume 107, Issue 2, June 2020, Pages 489–496

If I understand correctly, the paper claims that the marginal likelihood is similar to a very exhaustive cross-validation procedure, where you consider all possible training and test set splits (so also training sets of 1 sample) and where you use the posterior predictive density as a loss function/scoring rule. This could be seen as a disadvantage of marginal likelihood maximization because you would also take "weird" cross-validation partitions such as 1 training sample and N-1 test samples into account to estimate the hyper parameters

Improve this answer
$\endgroup$
-1
$\begingroup$

If you didn't have the other parameters $k$, then EB is identical to CV except that you don't have to search. You say that you are integrating out $k$ in both CV and EB. In that case, they are identical.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.