Metropolis-Hastings exercise with Cauchy and normal distributions [self-study]

Question

I have the following exercise to solve and would appreciate some help.

Consider a linear regression model $y = X\beta + \varepsilon$, where $y = (y_1,...,y_T)'$, $X = (x_1,...,x_T)$, $x_t$ is a single explanatory variable, $\beta$ is a scalar parameter, and $\varepsilon \sim N(0,1)$.

The likelihood function is given by$$p(y|\beta) = (\frac{1}{\sqrt{2 \pi}})^{T} \exp(-\frac{1}{2} (y-X\beta)'(y-X\beta))$$A Cauchy prior distribution is taken as prior on $\beta$: $\beta \sim CAU(0, \gamma)$ with density function:$$p(\beta)={1}\bigg/{\pi \gamma(1+\frac{\beta^2}{\gamma^2})}$$

To obtain the posterior we use Metropolis-Hastings algorithm, where a candidate density for $\beta$ is a normal density with mean $\hat\beta=(X'X)^{-1}(X'y)$ and covariance $(X'X)^{-1}$.

In italics I have put my answers.

a) Is the proposed MH sampler a random walk sampler or an independent sampler or neither of the two?

At this point I don't know how to assess it.

I know that if we have a candidate density $g(\beta|\beta^{(m)})$ it holds that $g(\beta|\beta^{(m)}) = g(\beta)$, where $\beta^{(m)}$ is the previous draw, for independence sampler.

For random walk sampler we have $\beta^{*}=\beta^{(m)}+\eta$, where $\beta^{*}$ is a value simulated from candidate density, but I can't see how to apply that knowledge for this case.

How, in general, can we distinguish independence sampler from RW sampler? In this case, how can I evaluate whether a draw from normal distribution depends on the previous draw?

b) Express the acceptance probability of the MH sampler in terms of the prior density only.

First we need to simulate $\beta^{(m)}$ from $g(\beta|\beta^{(m)})$ and calculate the acceptance-rejection probability:

$$\alpha=\min\Big( \frac{p(\beta^{*|}|y) g(\beta^{(m)}|\beta^{(*)})}{p(\beta^{(m)}|y)g(\beta^{(*)}|\beta^{(m)})}, 1 \Big) = \min\Big( \frac{p(\beta^{*|}|y)}{p(\beta^{(m)}|y)}, 1 \Big)$$

Since the candidate density is symmetric the terms cancel out. Based on this term I think that it is a random walk sampler. Is it the case that random walk sampler always assumes symmetric distribution as candidate density?

c) Prove that one accepts all candidate draws for a very large values of $\gamma$.

I have not idea how to start this proof. Cauchy distribution does not have a defined variance term and my only idea was to show that as we have small variance we accept all candidates.

d) Once we have valid draws $\beta^{(m)}$ for $m=1,...,M$ from the posterior distribution of $\beta$ how can we use these draws to generate draws from the predictive density $p(\exp(y_{T+1})|X,x_{T+1},y)?$

I came up with the following solution:

$$\alpha=\min \Big( \frac{p(\exp(y_{T+1})|X, x_{T+1}, y, \beta^{*})p(\beta^{(m)}|y)}{p(\exp(y_{T+1})|X, x_{T+1}, y, \beta^{(m)})p(\beta^{*}|y)},1 \Big)$$

Answer 1

1. An independent proposal does not depend on the previous realisation of the Markov chain, while a random walk proposal does in a symmetric way. Therefore when making a proposal from the $\mathcal N(\hat\beta,1/X'X)$ distribution, there is independence from the previous realisation (since $\hat\beta$ is fixed).

2. The generic acceptance probability is $$\alpha=\min\left( \frac{p(\beta^{*}|y) g(\beta^{(m)}|\beta^{(*)})}{p(\beta^{(m)}|y)g(\beta^{(*)}|\beta^{(m)})}, 1 \right)$$ which, in this case, is $$\alpha=\min\left( \frac{p(\beta^{*}|y) g(\beta^{(m)}|\hat\beta)}{p(\beta^{(m)}|y)g(\beta^{(*)}|\hat\beta)}, 1 \right)$$ and there is no cancellation for symmetry. However, $$p(y|\beta) \propto \exp\left\{- \frac{1}{2} ||y-\hat y||^2- \frac{1}{2} ((X'X)(\beta-\hat\beta)^2\right\}$$ and hence $p(y|\beta)\propto g(\beta|\hat\beta)$, meaning $$\alpha=\min\left( \frac{\pi(\beta^{*})}{\pi(\beta^{(m)})}, 1 \right)$$

3. Therefore if $\gamma$ is large, $\pi(\beta)$ is almost constant and the acceptance probability about equal to one.

4. If $y_{t+1}|\beta,x_{t+1}\sim\mathcal N(\beta x_{t+1},1)$, the posterior predictive distribution of $y_{t+1}$ is obtained by simulating $\beta^*$ from the posterior, i.e., sampling a term in the Markov chain, and simulating $y_{y+1}$ given $\beta^*$. The value of $z_{t+1}=\exp(y_{t+1})$ follows. (Do not use the notation $p(\exp(y_{t+1})|\ldots)$ since this is most likely incorrect, depending on the meaning of $p(\cdot|\cdot)$.