MCMC for an explicitly uncomputable prior?

Question

I am trying to sample from a posterior distribution and I only have an explicit formula for likelihood but I can sample from the prior distribution. How can I sample from the posterior distribution with such a restriction. Is there any specific method?

After seeing the answers I've decided to write my exact question to clarify stuff: Its about learning hyper-parameters $\alpha$ and $\beta$ and parameters $\theta_i$ in the following case:

$\alpha$ and $\beta$ are uniformly chosen from the perimeter of a square by following vertices: $(0,0),(0,1),(1,0),(1,1)$. Now $\theta_i$ is uniformly chosen from this line. $\theta_i$ it self is the parameter for data $y_i\sim\text{Bin}(n_i,\theta_i)$.

In my first attempt, and maybe being foolish I wrote a neat vectored algorithm which would sample from $p(\alpha,\beta,\theta)$ where $\theta=(\theta_1,\theta_2,...)$. But afterwards I realised that it is hardly related to sampling from $p(y|\alpha,\beta,\theta)$ maybe as a result of the answers here.

So what I am doing now is that I ignored the whole sampling algorithm I had for the joint priors. To solve the problem is to make a MC random-walk on parameter space $(\alpha,\beta)$ and sub-sampling from it (according to discussion on another question of mine in each step), then sampling from $p(\theta|\alpha,\beta)$ and then calculating the likelihood and then test the new sample according to Metropolis Hasting! I am not even sure this is correct but after my studies, this is the what I can think of!

Answer 1

Can you sample from the conditional distribution of $X\mid\Theta$? If you can, try using ABC to sample (approximately) from the posterior. The ABC rejection algorithm does not use the value of the prior density at each candidate point.

Answer 2

One specific method is importance sampling. The key slide in the link is slide 3.

In this case, you'd:

1. Generate a large random sample from your prior, let us denote it $\theta_i, i = 1, \dots, N$.

2. Each element of that sample will have associated with it a value of the likelihood function, let us say $l_i$. Calculate them.

3. We can then form resampling acceptance probabilities $p_i = l_i / \max l_i$.

4. Generate your posterior sample as follows:

   a) For each $j = 1, \dots,$ some large $M$, select some index $k$ uniformly from $\{1,\dots,N\}$.
   b) Generate $u \sim \text{U}(0,1)$.
   c) If $u < p_k$, then $\theta_k$ is put into your posterior sample. Otherwise, go to the next $j$, and nothing is put into your posterior sample.

Of course, if you can't generate a large random sample from the prior, or you have a few relatively large values of the likelihood and a lot of very small ones (which would happen if your posterior is very concentrated with respect to the prior), you won't get very good results. No panaceas here, I'm afraid! But this method works quite well in many cases.

Answer 3

How high dimensional is your state space? If it's a univariate problem I would suggest slice sampling. If it's higher dimensional you might be able to use slice sampling within Gibbs sampling still. If not, the other suggestions of ABC or importance sampling (some versions of ABC use importance sampling as an inner loop also), may be your best bet if there really isn't any additional structure you can exploit. As @jbowman says however, if your prior and posterior are very mismatched these methods will struggle.