What is the interpretation of this modified Metropolis algorithm? Modified Metropolis-Hastings
Consider a model with parameters $\theta = (\alpha, \gamma)$ and consider a modified Metropolis-Hastings algorithm which can be summarized (with brevity) as follows.

*

*Propose a new value $\alpha_\star$

*Conditional on $\alpha_\star$, fix $\gamma_\star$ according to some estimation rule (e.g. MLE or MAP)

*Calculate MH acceptance probability and choose whether or not to accept $\theta_\star = (\alpha_\star, \gamma_\star)$.


An Example
To help clarify, we can consider an extremely simple example. Let $X_1, X_2, \ldots X_{n}$ be iid samples from a $N(\alpha, \gamma)$ distribution. Let $\pi(\alpha, \gamma, {\bf X})$ denote the posterior distribution.

*

*Initialize $\alpha^0$.

*Set $\gamma^0 = \arg\max_{\gamma}\left\{ \pi(\alpha^0, \gamma, {\bf X})\right\}$

*Compute $\pi^0 = \pi(\alpha^0, \gamma^0, {\bf X})$
For $t = 1, 2, \ldots T$

*

*Propose $\alpha_\star$ (using a proposal, symmetric about $\alpha^{t-1}$ for simplicity).

*Set $\gamma_\star = \arg\max_{\gamma }\left\{ \pi(\alpha_\star, \gamma, {\bf X})\right\}$

*Calculate $\pi_\star = \pi(\alpha_\star, \gamma_\star, {\bf X})$

*Set $\alpha^t = \alpha_\star, \gamma^t = \gamma_\star$ and $\pi^t = \pi_\star$ with probability $\max\left(\frac{\pi_\star}{\pi^{t-1}}, 1\right)$ and set $\alpha^t = \alpha^{t-1}$, $\gamma^t = \gamma^{t-1}$ and $\pi^t = \pi^{t-1}$ otherwise.


Discussion and Questions

*

*I have performed a simple simulation study using the simple example described above, and found that this modified approach performed similarly (slightly better, actually) in terms of inference for $\alpha$ than the standard Bayesian approach.


*It seems that this approach can be viewed as placing a prior distribution on $\theta = (\alpha, \gamma)$ which places all of the prior density on regions of the parameter space where $\gamma$ is maximized with respect to $\alpha$ (a manifold, I believe).


*I am looking at this for much more complicated problems than the simple example above. In particular, problems where it is very difficult to propose reasonable values of $\gamma$ and maintain a computationally tractable Metropolis-Hastings ratio.


*Questions: Is this a valid approach to Bayesian inference? Does this algorithm have a name? Is there a nice way to interpret this approach? Literature discussing this in any way is quite welcome.
 A: This Metropolis-Hastings algorithm is exact for the distribution over $\alpha$
$$\tilde \pi(\alpha| \mathbf x) \propto \pi(\alpha,\arg\max_\gamma\pi(\alpha,\gamma,\mathbf x),\mathbf x)$$
which is a form of profile posterior. It thus does not target $\pi(\alpha,\gamma|\mathbf x)$ and does not bring information about $\gamma$. The difference with the average
$$\pi(\alpha| \mathbf x) \propto \int \pi(\alpha,\gamma,\mathbf x)\,\text d\gamma$$
means that there is no accounting for the variability in $\gamma$. Plus, setting $\gamma$ to depend on $\mathbf x$ as well as $\alpha$ means a potential for overfitting.
To check the difference, I look at a Normal case:
$$\pi(\alpha,\gamma|\mathbf x)\propto
\alpha^{-n-3}\exp[\{-n(\bar x-\gamma)^2-s^2-\delta\gamma^2-\beta\}/2\alpha^2]$$
where
$$\pi(\alpha|\mathbf x)\propto\alpha^{-n-2}\exp[\{-\underbrace{(n^{-1}+\delta^{-1})^{-1}\bar x^2}_{=\dfrac{n\delta\bar x^2}{n+\delta}}-s^2-\beta\}/2\alpha^2]$$
and
$$\gamma^\star(\alpha,\bar x)=\frac{n\bar x}{n+\delta}$$
implying
$$\tilde\pi(\alpha|\mathbf x)\propto\alpha^{-n-3}\exp[\{-n^2\delta\bar x^2/(n+\delta)^2-s^2-\beta\}/2\alpha^2]$$
which shows the difference between both "posteriors". The pseudo-posterior $\tilde\pi(\alpha|\mathbf x)$ is more concentrated near zero, reflecting upon the extra confidence on $\gamma$ brought by using the "best" value of $\gamma$.
