What is the interpretation of this modified Metropolis algorithm?

Question

Modified Metropolis-Hastings

Consider a model with parameters $\theta = (\alpha, \gamma)$ and consider a modified Metropolis-Hastings algorithm which can be summarized (with brevity) as follows.

1. Propose a new value $\alpha_\star$
2. Conditional on $\alpha_\star$, fix $\gamma_\star$ according to some estimation rule (e.g. MLE or MAP)
3. Calculate MH acceptance probability and choose whether or not to accept $\theta_\star = (\alpha_\star, \gamma_\star)$.

---

An Example

To help clarify, we can consider an extremely simple example. Let $X_1, X_2, \ldots X_{n}$ be iid samples from a $N(\alpha, \gamma)$ distribution. Let $\pi(\alpha, \gamma, {\bf X})$ denote the posterior distribution.

1. Initialize $\alpha^0$.
2. Set $\gamma^0 = \arg\max_{\gamma}\left\{ \pi(\alpha^0, \gamma, {\bf X})\right\}$
3. Compute $\pi^0 = \pi(\alpha^0, \gamma^0, {\bf X})$

For $t = 1, 2, \ldots T$

1. Propose $\alpha_\star$ (using a proposal, symmetric about $\alpha^{t-1}$ for simplicity).
2. Set $\gamma_\star = \arg\max_{\gamma }\left\{ \pi(\alpha_\star, \gamma, {\bf X})\right\}$
3. Calculate $\pi_\star = \pi(\alpha_\star, \gamma_\star, {\bf X})$
4. Set $\alpha^t = \alpha_\star, \gamma^t = \gamma_\star$ and $\pi^t = \pi_\star$ with probability $\max\left(\frac{\pi_\star}{\pi^{t-1}}, 1\right)$ and set $\alpha^t = \alpha^{t-1}$, $\gamma^t = \gamma^{t-1}$ and $\pi^t = \pi^{t-1}$ otherwise.

---

Discussion and Questions

• I have performed a simple simulation study using the simple example described above, and found that this modified approach performed similarly (slightly better, actually) in terms of inference for $\alpha$ than the standard Bayesian approach.

• It seems that this approach can be viewed as placing a prior distribution on $\theta = (\alpha, \gamma)$ which places all of the prior density on regions of the parameter space where $\gamma$ is maximized with respect to $\alpha$ (a manifold, I believe).

• I am looking at this for much more complicated problems than the simple example above. In particular, problems where it is very difficult to propose reasonable values of $\gamma$ and maintain a computationally tractable Metropolis-Hastings ratio.

• Questions: Is this a valid approach to Bayesian inference? Does this algorithm have a name? Is there a nice way to interpret this approach? Literature discussing this in any way is quite welcome.

Answer 1

This Metropolis-Hastings algorithm is exact for the distribution over $\alpha$ $$\tilde \pi(\alpha| \mathbf x) \propto \pi(\alpha,\arg\max_\gamma\pi(\alpha,\gamma,\mathbf x),\mathbf x)$$ which is a form of profile posterior. It thus does not target $\pi(\alpha,\gamma|\mathbf x)$ and does not bring information about $\gamma$. The difference with the average $$\pi(\alpha| \mathbf x) \propto \int \pi(\alpha,\gamma,\mathbf x)\,\text d\gamma$$ means that there is no accounting for the variability in $\gamma$. Plus, setting $\gamma$ to depend on $\mathbf x$ as well as $\alpha$ means a potential for overfitting.

To check the difference, I look at a Normal case: $$\pi(\alpha,\gamma|\mathbf x)\propto \alpha^{-n-3}\exp[\{-n(\bar x-\gamma)^2-s^2-\delta\gamma^2-\beta\}/2\alpha^2]$$ where $$\pi(\alpha|\mathbf x)\propto\alpha^{-n-2}\exp[\{-\underbrace{(n^{-1}+\delta^{-1})^{-1}\bar x^2}_{=\dfrac{n\delta\bar x^2}{n+\delta}}-s^2-\beta\}/2\alpha^2]$$ and $$\gamma^\star(\alpha,\bar x)=\frac{n\bar x}{n+\delta}$$ implying $$\tilde\pi(\alpha|\mathbf x)\propto\alpha^{-n-3}\exp[\{-n^2\delta\bar x^2/(n+\delta)^2-s^2-\beta\}/2\alpha^2]$$ which shows the difference between both "posteriors". The pseudo-posterior $\tilde\pi(\alpha|\mathbf x)$ is more concentrated near zero, reflecting upon the extra confidence on $\gamma$ brought by using the "best" value of $\gamma$.