Skip to main content
2 of 5
more attractive title perhaps?
Memming
  • 1.7k
  • 13
  • 23

Cross-validation vs empirical Bayes for estimating hyperparameters

Given a hierarchical model $p(x|k,\theta)$, I want a two stage process to fit the model. First, fix some a handful of hyperparameters $\theta$, and then do Bayesian inference on the rest of the parameters $k$. For fixing the hyperparameters I am considering two options.

  1. Use Empirical Bayes (EB) and maximize the marginal likelihood $p(\mbox{all data}|\theta)$ (integrating out the rest of the model which contains high dimensional parameters).
  2. Use Cross Validation (CV) techniques such as $k$-fold cross validation to choose $\theta$ that maximizes the likelihood $p(\mbox{test data}|\mbox{training data}, \theta)$.

The advantage of EB is that I can use all data at once, while for CV I need to (potentially) compute the model likelihood multiple times and search for $\theta$. The performance of EB and CV are comparable in many cases, and often EB is faster to estimate.

Question: Is there a theoretical foundation that links the two (say, EB and CV are the same in the limit of large data)? Or links EB to some generalizability criterion such as empirical risk? Can someone point to a good reference material?

Memming
  • 1.7k
  • 13
  • 23