I am reading slides on Tutorial for Bayesian Optimization [42/45] and came across this
"Covariance hyperparameters are often optimized rather than marginalized, typically in the name of convenience and efficiency"
I am trying to understand the significance of marginalization over optimization of covariance hyperparameters. Optimization is entirely different concept then marginalization, so what is the link between the two?
Let's say we have a density $f(x; \theta, \eta)$ and $\theta$ is a parameter of interest but $\eta$ is a nuisance parameter, i.e. we need to know its value to evaluate the density but we don't actually care about it.
We want to somehow get rid of $\eta$ and end up with something of the form $g(x; \theta)$.
Integration and optimization are two common approaches to this. For integration we could put a prior on $\eta$, say $\pi(\eta)$, and obtain $g_I$ via $$ g_I(x;\theta) = \int f(x;\theta,\eta)\pi(\eta)\,\text d\eta. $$ Intuitively, evaluating $g_I$ is like averaging $f$ over all possible values of $\eta$, weighted by their likeliness.
But another option is to plug just one value in for $\eta$. We could find $$ \hat\eta(x, \theta) = \underset{\eta}{\text{argmax}}\, f(x; \theta, \eta) $$ and then get $$ g_p(x; \theta) = f(x; \theta, \hat\eta(x, \theta)). $$
We could arrive at this by using a Dirac delta at $\hat\eta(x, \theta)$ and integrating against that, i.e. we could use a prior (that would need to depend on $x$ and $\theta$, so philosophically not really a prior) that puts a probability of $1$ on this maximum, and then $$ g_p(x; \theta) = \int f(x; \theta, \eta) \delta_{\hat\eta}(\eta)\,\text d\eta. $$
(I'm using a subscript $p$ since this is called "profiling")
So this is one way to see how they connect. Optimization is putting all our eggs in one basket and thinking that we can represent $f$ well by just using the most likely value of $\eta$, while integration is instead considering all the possible values but we're weighting according to our believed likeliness. Optimization is usually way easier computationally which is a big part of the appeal, although integration can work better. My answer here gives an example of that, and there's also a generally interesting discussion: MLE: Marginal vs Full Likelihood
This paper by Berger et al. is also interesting: https://www2.stat.duke.edu/~berger/papers/brunero.pdf
