I am currently reading Rasmussen & Ghahramani's paper on Infinite Mixtures of Gaussian Process Experts. (you can find it at this link)
Unfortunately, I lack experience with understanding how complex statistical models are learnt, especially when multiple techniques are combined. As a result, there are a couple of points which I do not understand about the algorithm specified in the paper.
I've listed out the algorithm's steps at the bottom for convenience. My questions are:
- Step 3 is to "Do Hybrid Monte Carlo". Given that Monte Carlo is sampling, how is this different from doing "Sample", as specified in steps 5 and 6?
- More on steps 5 and 6, do the steps imply that each parameter is sampled once per algorithm iteration (one cycle from steps 2 to 6), or does this imply sampling multiple times and then taking the sample mean to compute the expectation of the parameter? (if yes, why are we taking the expectation in the first place?)
- Step 4 asks to "Optimize" the hyper-hypers - does this imply something like maximum likelihood estimation, and in the first place what exactly is the quantity that we are aiming to optimize here?
- Step 6 has a footnote specifying that the proposal is asymmetric, even though it is a Gaussian proposal (aren't Gaussian proposals symmetric?). The kernel widths are positive, so I suppose the proposal is asymmetric to prevent sampling from outside of the positive reals - but how is this done? Is the "Gaussian proposal" actually supposed to be log normal instead?
- Step 6's footnote also mentions that the Gaussian fit uses the derivative and Hessian of the log posterior - what does this mean?
Your answer is of course very much appreciated even if you are addressing not all of the points above - any clarification at all would already be very helpful!
As mentioned above, the algorithm for learning the data, as outlined by the paper, is:
- Initialize indicator variables $c_i$ to a single value.
- Do a Gibbs sampling sweep over all indicators.
- Do Hybrid Monte Carlo for hyperparameters of the GP covariance function, $v_0; v_1; w_d$, for each expert in turn.
- Optimize the hyper-hypers, $a$ & $b$, for each of the variance parameters.
- Sample the Dirichlet process concentration parameter, $\alpha$ using ARS.
- Sample the gating kernel widths $\phi$, using the Metropolis method to sample from the pseudo-posterior with a Gaussian proposal fit at the current $\phi$.
- Repeat from 2 until the Markov chain has adequately sampled the posterior.