1
$\begingroup$

I am trying to sample from a posterior distribution and I only have an explicit formula for likelihood but I can sample from the prior distribution. How can I sample from the posterior distribution with such a restriction. Is there any specific method?

After seeing the answers I've decided to write my exact question to clarify stuff: Its about learning hyper-parameters $\alpha$ and $\beta$ and parameters $\theta_i$ in the following case:

$\alpha$ and $\beta$ are uniformly chosen from the perimeter of a square by following vertices: $(0,0),(0,1),(1,0),(1,1)$. Now $\theta_i$ is uniformly chosen from this line. $\theta_i$ it self is the parameter for data $y_i\sim\text{Bin}(n_i,\theta_i)$.

In my first attempt, and maybe being foolish I wrote a neat vectored algorithm which would sample from $p(\alpha,\beta,\theta)$ where $\theta=(\theta_1,\theta_2,...)$. But afterwards I realised that it is hardly related to sampling from $p(y|\alpha,\beta,\theta)$ maybe as a result of the answers here.

So what I am doing now is that I ignored the whole sampling algorithm I had for the joint priors. To solve the problem is to make a MC random-walk on parameter space $(\alpha,\beta)$ and sub-sampling from it (according to discussion on another question of mine in each step), then sampling from $p(\theta|\alpha,\beta)$ and then calculating the likelihood and then test the new sample according to Metropolis Hasting! I am not even sure this is correct but after my studies, this is the what I can think of!

$\endgroup$
  • $\begingroup$ Just out of curiosity, how did you end up with an uncomputable prior that you can sample from? Is it an output from another MCMC process, by any chance? $\endgroup$ – jbowman Oct 28 '13 at 1:31
  • $\begingroup$ @jbowman, I will state my exact question. I think that would clarify most of the things. $\endgroup$ – Cupitor Oct 28 '13 at 15:13
  • $\begingroup$ @jbowman, still I wonder what would be the answer in general case for such priors. $\endgroup$ – Cupitor Oct 28 '13 at 15:32
2
$\begingroup$

Can you sample from the conditional distribution of $X\mid\Theta$? If you can, try using ABC to sample (approximately) from the posterior. The ABC rejection algorithm does not use the value of the prior density at each candidate point.

$\endgroup$
  • $\begingroup$ Thank you. If by $X|\theta$, you mean the likelihood, as I said I can compute the likelihood explicitly. Is it the case? $\endgroup$ – Cupitor Oct 27 '13 at 22:00
  • 3
    $\begingroup$ But those are different things. One thing is compute the value of the likelihood for some value of the parameter (this gives you a real number). Another thing is, given a value of the parameter simulated from the prior, to simulate a synthetic $X$ from from the conditional distribution (ABC needs this). The duality likelihood x sampling distribution is important here. $\endgroup$ – Zen Oct 28 '13 at 0:56
  • $\begingroup$ Sorry! My mistake! I see what you mean. I cannot sample explicitly from $X|\Theta$ but I can sample from it using another MCMC. $\endgroup$ – Cupitor Oct 28 '13 at 16:10
2
$\begingroup$

One specific method is importance sampling. The key slide in the link is slide 3.

In this case, you'd:

  1. Generate a large random sample from your prior, let us denote it $\theta_i, i = 1, \dots, N$.

  2. Each element of that sample will have associated with it a value of the likelihood function, let us say $l_i$. Calculate them.

  3. We can then form resampling acceptance probabilities $p_i = l_i / \max l_i$.

  4. Generate your posterior sample as follows:

    a) For each $j = 1, \dots,$ some large $M$, select some index $k$ uniformly from $\{1,\dots,N\}$.
    b) Generate $u \sim \text{U}(0,1)$.
    c) If $u < p_k$, then $\theta_k$ is put into your posterior sample. Otherwise, go to the next $j$, and nothing is put into your posterior sample.

Of course, if you can't generate a large random sample from the prior, or you have a few relatively large values of the likelihood and a lot of very small ones (which would happen if your posterior is very concentrated with respect to the prior), you won't get very good results. No panaceas here, I'm afraid! But this method works quite well in many cases.

$\endgroup$
1
$\begingroup$

How high dimensional is your state space? If it's a univariate problem I would suggest slice sampling. If it's higher dimensional you might be able to use slice sampling within Gibbs sampling still. If not, the other suggestions of ABC or importance sampling (some versions of ABC use importance sampling as an inner loop also), may be your best bet if there really isn't any additional structure you can exploit. As @jbowman says however, if your prior and posterior are very mismatched these methods will struggle.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.