0
$\begingroup$

Bayes' rule tells us that,

$P(h|d) = \frac{ P(d|h) P(h) }{ \sum_i P(d|h_i) P(h_i) }$.

Let's say we have four hypotheses: $h_1$, $h_2$, $h_3$, and $h_4$. The likelihood over hypotheses looks like this:

$P(d|h) = [0.1, 0.2, 0.3, 0.4]$

and the prior looks like this:

$P(h) = [0.4, 0.3, 0.2, 0.1]$

Multiplying, we find that the unnormalized posterior is:

$P(h|d) \propto [0.04, 0.06, 0.06, 0.04]$

Now, let's say that we want to compare what happens when we make the prior stronger by squaring it, so our prior is now:

$P(h)^2 = [0.16, 0.09,0.04,0.01]$

and the unnormalized posterior is now:

$P(h|d) \propto [0.016,0.018,0.012,0.004]$

On the face of it, it appears that the posterior probabilities for all four hypotheses have gotten smaller. However, if we had normalized the posteriors, we would have found that the posterior probability has actually gotten larger for some hypotheses ($h_1$ and $h_2$) and smaller for others ($h_3$ and $h_4$). For example, $P(h_1|d)$ increases from 0.2 to 0.32 when normalized, although this isn't immediately evident when looking at the unnormalized posteriors above.

Let's say that the hypothesis space is very large, such that it is infeasible to calculate the denominator in Bayes' rule (the normalizing constant). Is there a method to correct the unnormalized posterior to account for the weight $w$ that has been applied to the prior? We don't care about getting the true value of the posterior, we just want to account for the fact that the values will generally be smaller as the weight gets larger, so that we can make fair comparisons.

For example, let's say we were only considering $h_1$, and we want to compare what happens when $w=2$ and $w=3$. We find that:

$P(h_1|d,w=2) \propto 0.016$

and

$P(h_1|d,w=3) \propto 0.0064$

Without summing over the hypothesis space, can we correct these values to take account of $w$? In this case the correction should tell us that $h_1$ is actually 1.369863 times more probable when $w = 3$ compared to when $w = 2$.

It's quite likely that what I'm asking for is mathematically impossible, but it's not obvious to me why.

$\endgroup$
  • $\begingroup$ What's the dimension of $\theta$? $\endgroup$ – mef Feb 20 '18 at 18:10
  • $\begingroup$ I'm not sure, but I think we could just consider it to be one dimensional. If some more background helps... a hypothesis is any possible partition of the numbers 1–64 into 1–4 sets (in fact, a hypothesis is a partition of a 2D space, an 8x8 grid). This is for an agent-based model, in which a Bayesian agent has to select the best hypothesis to explain a dataset (limited evidence about which set some of the numbers go into). I can provide more background if it's helpful, but I'm also just curious if something like this can be done for one dimension. $\endgroup$ – Jon Carr Feb 20 '18 at 19:09
  • $\begingroup$ How do you choose which ones to compare (out of the very large number of possibilities)? $\endgroup$ – mef Feb 22 '18 at 15:49
  • $\begingroup$ @mef I select a hypothesis by Metropolis-Hastings, which doesn't require posterior normalization because it only cares about the ratios between probabilities. Now I want to vary how the prior is weighted to see which weight setting results in the highest posterior. The problem is, the best weight will always be 0 because $w=0$ will always yield the biggest unnormalized probabilities, but not necessarily the biggest normalized probabilities. $\endgroup$ – Jon Carr Feb 22 '18 at 20:33
  • $\begingroup$ The draws provide approximations to the normalized probabilities. The number of times you draw a given 'hypothesis" divided by the number of draws is an approximation to the normalized probability. To get more accuracy, take more draws. After you change the prior, resample. Then you can compare. $\endgroup$ – mef Feb 22 '18 at 20:39
0
$\begingroup$

If you can sample from the posterior (via MCMC for example), then you can approximate the normalized posterior probabilities. The number of times you draw a given "hypothesis" divided by the number of draws is an approximation to the normalized probability of that hypothesis. To get more accuracy, take more draws. After you change the prior, sample from the new posterior. Then you can compare probabilities (for hypotheses) for each of the priors chosen.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.