3
$\begingroup$

Consider a heteroskedastic model of the form, $y_i|x_i \sim \mathcal{N}\left(x_i, \text{exp}\{\boldsymbol\beta^\top\boldsymbol{x}\}\right)$ where $\boldsymbol{\beta}=\left[\beta_0,\beta_1\right]$ and $\boldsymbol{x}=\left[1,x_i\right]$. $x_i$ and $\boldsymbol{\beta}$ must be estimated from the data.

We set a multivariate normal prior on $\boldsymbol{\beta}, \boldsymbol{\beta}\sim \mathcal{N}\left(0,I\right)$ and try to derive the posterior for $\boldsymbol{\beta}, p\left(\boldsymbol{\beta}|y,\boldsymbol{x}\right)$ for the purpose of sampling. It is not possible to get a posterior in closed form, so we must resort to Metropolis-Hastings. For M-H to work we sample a new value $\boldsymbol{\beta}^*$ from a multivariate normal proposal density of the form $\boldsymbol{\beta}^*\sim\mathcal{N}\left(\boldsymbol{\beta}^{t},\Sigma\right)$, where $\boldsymbol{\beta}^t$ is the value from the current iteration.

The question I have is, how does one choose $\Sigma$, the proposal variance, given that there are no predictors only latent variables.

This problem is similar to Poisson regression except that unlike Poisson regression problem, we have unobserved or latent variables instead of predictors. Hence, proposal variance cannot be estimated from the data.

Any references are appreciated.

| cite | improve this question | |
$\endgroup$
2
$\begingroup$

Under $d \to \infty$ optimal acceptance rate is 0.234. The reference is Roberts et al 1997. Although after skimming the paper it appears to have been a result originally derived in an earlier paper. In any case this is the often cited paper which has some nice results and useful heuristics. Given that in your case you have $d=2$ it is unclear how relevant these results may be but are worth trying out empirically on your sampler for a first pass. For $d=1$ case it seems (see Roberts and Rosenthal) that the optimal acceptance rate is around 0.44 so you may want to experiment with some values of $\Sigma$ that get you into the region $(0.233, 0.44)$ or maybe even a bit larger of an interval just to be on the safe side.
For simplicity you can probably use a diagonal $\Sigma$ for a first pass.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Your point about acceptance rate is well taken. However, my question is how do I choose $\Sigma$. Once idea similar to your suggestion is to choose a diagonal $\Sigma = \sigma^2\text{I}$. However, I still need to tune the constant $\sigma^2$. I am looking for something on the lines of adaptive M-H that adjusts the tuning constant to get an acceptance rate in the interval $\left(0.233,0.44\right)$ $\endgroup$ – Kumar Jan 12 '18 at 15:52
  • $\begingroup$ in practice the $\sigma^2$ (proposal variance) is often chosen from trying several values and comparing the acceptance rates. Then choose one value of $\sigma^2$ that gives you a reasonable acceptance rate in the interval I suggested. Have you tried a static proposal variance? I would try it first to see how well it does before trying something more complicated like adaptive M-H. $\endgroup$ – Lucas Roberts Jan 12 '18 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.