Relation between Optimization of Hyperparameters and Marginalization (Bayesian Optimization) I am reading slides on Tutorial for Bayesian Optimization [42/45] and came across this
"Covariance hyperparameters are often optimized rather than
marginalized, typically in the name of convenience and efficiency"
I am trying to understand the significance of marginalization over optimization of covariance hyperparameters. Optimization is entirely different concept then marginalization, so what is the link between the two?
 A: Let's say we have a density $f(x; \theta, \eta)$ and $\theta$ is a parameter of interest but $\eta$ is a nuisance parameter, i.e. we need to know its value to evaluate the density but we don't actually care about it.
We want to somehow get rid of $\eta$ and end up with something of the form $g(x; \theta)$. 
Integration and optimization are two common approaches to this. For integration we could put a prior on $\eta$, say $\pi(\eta)$, and obtain $g_I$ via
$$
g_I(x;\theta) = \int f(x;\theta,\eta)\pi(\eta)\,\text d\eta.  
$$
Intuitively, evaluating $g_I$ is like averaging $f$ over all possible values of $\eta$, weighted by their likeliness. 
But another option is to plug just one value in for $\eta$. We could find
$$
\hat\eta(x, \theta) = \underset{\eta}{\text{argmax}}\, f(x; \theta, \eta)
$$
and then get 
$$
g_p(x; \theta) = f(x; \theta, \hat\eta(x, \theta)).
$$
We could arrive at this by using a Dirac delta at $\hat\eta(x, \theta)$ and integrating against that, i.e. we could use a prior (that would need to depend on $x$ and $\theta$, so philosophically not really a prior) that puts a probability of $1$ on this maximum, and then
$$
g_p(x; \theta) = \int f(x; \theta, \eta) \delta_{\hat\eta}(\eta)\,\text d\eta.
$$
(I'm using a subscript $p$ since this is called "profiling")
So this is one way to see how they connect. Optimization is putting all our eggs in one basket and thinking that we can represent $f$ well by just using the most likely value of $\eta$, while integration is instead considering all the possible values but we're weighting according to our believed likeliness. Optimization is usually way easier computationally which is a big part of the appeal, although integration can work better. My answer here gives an example of that, and there's also a generally interesting discussion: MLE: Marginal vs Full Likelihood 
This paper by Berger et al. is also interesting: https://www2.stat.duke.edu/~berger/papers/brunero.pdf
