I have the following exercise to solve and would appreciate some help.

Consider a linear regression model $y = X\beta + \varepsilon$, where $y = (y_1,...,y_T)'$, $X = (x_1,...,x_T)$, $x_t$ is a single explanatory variable, $\beta$ is a scalar parameter, and $\varepsilon \sim N(0,1)$.

The likelihood function is given by$$p(y|\beta) = (\frac{1}{\sqrt{2 \pi}})^{T} \exp(-\frac{1}{2} (y-X\beta)'(y-X\beta))$$A Cauchy prior distribution is taken as prior on $\beta$: $\beta \sim CAU(0, \gamma)$ with density function:$$p(\beta)={1}\bigg/{\pi \gamma(1+\frac{\beta^2}{\gamma^2})}$$

To obtain the posterior we use Metropolis-Hastings algorithm, where a candidate density for $\beta$ is a normal density with mean $\hat\beta=(X'X)^{-1}(X'y)$ and covariance $(X'X)^{-1}$.

In italics I have put my answers.

a) Is the proposed MH sampler a random walk sampler or an independent sampler or neither of the two?

At this point I don't know how to assess it.

I know that if we have a candidate density $g(\beta|\beta^{(m)})$ it holds that $g(\beta|\beta^{(m)}) = g(\beta)$, where $\beta^{(m)}$ is the previous draw, for independence sampler.

For random walk sampler we have $\beta^{*}=\beta^{(m)}+\eta$, where $\beta^{*}$ is a value simulated from candidate density, but I can't see how to apply that knowledge for this case.

How, in general, can we distinguish independence sampler from RW sampler? In this case, how can I evaluate whether a draw from normal distribution depends on the previous draw?

b) Express the acceptance probability of the MH sampler in terms of the prior density only.

First we need to simulate $\beta^{(m)}$ from $g(\beta|\beta^{(m)})$ and calculate the acceptance-rejection probability:

$$\alpha=\min\Big( \frac{p(\beta^{*|}|y) g(\beta^{(m)}|\beta^{(*)})}{p(\beta^{(m)}|y)g(\beta^{(*)}|\beta^{(m)})}, 1 \Big) = \min\Big( \frac{p(\beta^{*|}|y)}{p(\beta^{(m)}|y)}, 1 \Big)$$

Since the candidate density is symmetric the terms cancel out. Based on this term I think that it is a random walk sampler. Is it the case that random walk sampler always assumes symmetric distribution as candidate density?

c) Prove that one accepts all candidate draws for a very large values of $\gamma$.

I have not idea how to start this proof. Cauchy distribution does not have a defined variance term and my only idea was to show that as we have small variance we accept all candidates.

d) Once we have valid draws $\beta^{(m)}$ for $m=1,...,M$ from the posterior distribution of $\beta$ how can we use these draws to generate draws from the predictive density $p(\exp(y_{T+1})|X,x_{T+1},y)?$

I came up with the following solution:

$$\alpha=\min \Big( \frac{p(\exp(y_{T+1})|X, x_{T+1}, y, \beta^{*})p(\beta^{(m)}|y)}{p(\exp(y_{T+1})|X, x_{T+1}, y, \beta^{(m)})p(\beta^{*}|y)},1 \Big)$$

  • $\begingroup$ The trouble is that you are using $\beta$ as a single notation for many different things. $\endgroup$
    – Xi'an
    Commented Jul 23, 2020 at 13:59
  • $\begingroup$ Note also that multidimensional notations are used for $X$ although it is unidimensional:$$X'y=\sum_t x_y y_t\qquad X'X=\sum_t x_t^2$$ $\endgroup$
    – Xi'an
    Commented Jul 30, 2020 at 17:46

1 Answer 1

  1. An independent proposal does not depend on the previous realisation of the Markov chain, while a random walk proposal does in a symmetric way. Therefore when making a proposal from the $\mathcal N(\hat\beta,1/X'X)$ distribution, there is independence from the previous realisation (since $\hat\beta$ is fixed).

  2. The generic acceptance probability is $$\alpha=\min\left( \frac{p(\beta^{*}|y) g(\beta^{(m)}|\beta^{(*)})}{p(\beta^{(m)}|y)g(\beta^{(*)}|\beta^{(m)})}, 1 \right)$$ which, in this case, is $$\alpha=\min\left( \frac{p(\beta^{*}|y) g(\beta^{(m)}|\hat\beta)}{p(\beta^{(m)}|y)g(\beta^{(*)}|\hat\beta)}, 1 \right)$$ and there is no cancellation for symmetry. However, $$p(y|\beta) \propto \exp\left\{- \frac{1}{2} ||y-\hat y||^2- \frac{1}{2} ((X'X)(\beta-\hat\beta)^2\right\}$$ and hence $p(y|\beta)\propto g(\beta|\hat\beta)$, meaning $$\alpha=\min\left( \frac{\pi(\beta^{*})}{\pi(\beta^{(m)})}, 1 \right)$$

  3. Therefore if $\gamma$ is large, $\pi(\beta)$ is almost constant and the acceptance probability about equal to one.

  4. If $y_{t+1}|\beta,x_{t+1}\sim\mathcal N(\beta x_{t+1},1)$, the posterior predictive distribution of $y_{t+1}$ is obtained by simulating $\beta^*$ from the posterior, i.e., sampling a term in the Markov chain, and simulating $y_{y+1}$ given $\beta^*$. The value of $z_{t+1}=\exp(y_{t+1})$ follows. (Do not use the notation $p(\exp(y_{t+1})|\ldots)$ since this is most likely incorrect, depending on the meaning of $p(\cdot|\cdot)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.