# Metropolis-Hastings exercise with Cauchy and normal distributions [self-study]

I have the following exercise to solve and would appreciate some help.

Consider a linear regression model $$y = X\beta + \varepsilon$$, where $$y = (y_1,...,y_T)'$$, $$X = (x_1,...,x_T)$$, $$x_t$$ is a single explanatory variable, $$\beta$$ is a scalar parameter, and $$\varepsilon \sim N(0,1)$$.

The likelihood function is given by$$p(y|\beta) = (\frac{1}{\sqrt{2 \pi}})^{T} \exp(-\frac{1}{2} (y-X\beta)'(y-X\beta))$$A Cauchy prior distribution is taken as prior on $$\beta$$: $$\beta \sim CAU(0, \gamma)$$ with density function:$$p(\beta)={1}\bigg/{\pi \gamma(1+\frac{\beta^2}{\gamma^2})}$$

To obtain the posterior we use Metropolis-Hastings algorithm, where a candidate density for $$\beta$$ is a normal density with mean $$\hat\beta=(X'X)^{-1}(X'y)$$ and covariance $$(X'X)^{-1}$$.

In italics I have put my answers.

a) Is the proposed MH sampler a random walk sampler or an independent sampler or neither of the two?

At this point I don't know how to assess it.

I know that if we have a candidate density $$g(\beta|\beta^{(m)})$$ it holds that $$g(\beta|\beta^{(m)}) = g(\beta)$$, where $$\beta^{(m)}$$ is the previous draw, for independence sampler.

For random walk sampler we have $$\beta^{*}=\beta^{(m)}+\eta$$, where $$\beta^{*}$$ is a value simulated from candidate density, but I can't see how to apply that knowledge for this case.

How, in general, can we distinguish independence sampler from RW sampler? In this case, how can I evaluate whether a draw from normal distribution depends on the previous draw?

b) Express the acceptance probability of the MH sampler in terms of the prior density only.

First we need to simulate $$\beta^{(m)}$$ from $$g(\beta|\beta^{(m)})$$ and calculate the acceptance-rejection probability:

$$\alpha=\min\Big( \frac{p(\beta^{*|}|y) g(\beta^{(m)}|\beta^{(*)})}{p(\beta^{(m)}|y)g(\beta^{(*)}|\beta^{(m)})}, 1 \Big) = \min\Big( \frac{p(\beta^{*|}|y)}{p(\beta^{(m)}|y)}, 1 \Big)$$

Since the candidate density is symmetric the terms cancel out. Based on this term I think that it is a random walk sampler. Is it the case that random walk sampler always assumes symmetric distribution as candidate density?

c) Prove that one accepts all candidate draws for a very large values of $$\gamma$$.

I have not idea how to start this proof. Cauchy distribution does not have a defined variance term and my only idea was to show that as we have small variance we accept all candidates.

d) Once we have valid draws $$\beta^{(m)}$$ for $$m=1,...,M$$ from the posterior distribution of $$\beta$$ how can we use these draws to generate draws from the predictive density $$p(\exp(y_{T+1})|X,x_{T+1},y)?$$

I came up with the following solution:

$$\alpha=\min \Big( \frac{p(\exp(y_{T+1})|X, x_{T+1}, y, \beta^{*})p(\beta^{(m)}|y)}{p(\exp(y_{T+1})|X, x_{T+1}, y, \beta^{(m)})p(\beta^{*}|y)},1 \Big)$$

• The trouble is that you are using $\beta$ as a single notation for many different things. Jul 23 '20 at 13:59
• Note also that multidimensional notations are used for $X$ although it is unidimensional:$$X'y=\sum_t x_y y_t\qquad X'X=\sum_t x_t^2$$ Jul 30 '20 at 17:46

## 1 Answer

1. An independent proposal does not depend on the previous realisation of the Markov chain, while a random walk proposal does in a symmetric way. Therefore when making a proposal from the $$\mathcal N(\hat\beta,1/X'X)$$ distribution, there is independence from the previous realisation (since $$\hat\beta$$ is fixed).

2. The generic acceptance probability is $$\alpha=\min\left( \frac{p(\beta^{*}|y) g(\beta^{(m)}|\beta^{(*)})}{p(\beta^{(m)}|y)g(\beta^{(*)}|\beta^{(m)})}, 1 \right)$$ which, in this case, is $$\alpha=\min\left( \frac{p(\beta^{*}|y) g(\beta^{(m)}|\hat\beta)}{p(\beta^{(m)}|y)g(\beta^{(*)}|\hat\beta)}, 1 \right)$$ and there is no cancellation for symmetry. However, $$p(y|\beta) \propto \exp\left\{- \frac{1}{2} ||y-\hat y||^2- \frac{1}{2} ((X'X)(\beta-\hat\beta)^2\right\}$$ and hence $$p(y|\beta)\propto g(\beta|\hat\beta)$$, meaning $$\alpha=\min\left( \frac{\pi(\beta^{*})}{\pi(\beta^{(m)})}, 1 \right)$$

3. Therefore if $$\gamma$$ is large, $$\pi(\beta)$$ is almost constant and the acceptance probability about equal to one.

4. If $$y_{t+1}|\beta,x_{t+1}\sim\mathcal N(\beta x_{t+1},1)$$, the posterior predictive distribution of $$y_{t+1}$$ is obtained by simulating $$\beta^*$$ from the posterior, i.e., sampling a term in the Markov chain, and simulating $$y_{y+1}$$ given $$\beta^*$$. The value of $$z_{t+1}=\exp(y_{t+1})$$ follows. (Do not use the notation $$p(\exp(y_{t+1})|\ldots)$$ since this is most likely incorrect, depending on the meaning of $$p(\cdot|\cdot)$$.