Bayesian probability > 1 -- is it possible?

Question

Running a crude model using Bayesian inference, I get some results > 1 (ie, more than 100% "certain") for some combinations of "evidence". For instance, for one bit of evidence the conditional probability of the null hypothesis is 0.85 while the marginal probability is 0.77. If the prior probability is 0.9, the computed posterior probability is 1.008. Which maybe could be ascribed to rounding error, except that the next bit of evidence raises the posterior probability to 1.34.

It stands to reason that, for a problem with two hypotheses, one would have a conditional probability less than the marginal probability and the other would be greater. So the resulting P(E|H) / P(E) multiplier would be > 1. So it's hard to see how such > 1 results can be avoided in the general case.

Is this just the way Bayesian inference works, or do I likely have an error somewhere in my calcs?

Data

One "evidence":
Total 6134 samples
Total actual positives 2845
Total actual negatives 3289
Test was true 1623 times
Test was true 465 times when the "gold standard" was positive
Test was true 1158 times when the "gold standard" was negative

conditional probability of positive hypothesis = 465/2845 = 0.1634
conditional probability of negative hypothesis = 1158/3289 = 0.3521
marginal probability of test being true = 1623/6134 = 0.2646
Bayes multiplier for the negative hypothesis = 0.3521 / 0.2646 = 1.3307

As can be seen, the multiplier is significantly > 1, and with several such tests back-to-back it seems hard to avoid probabilities > 1. (Of course, I suppose one can argue that the tests aren't truly independent, and that puts the fly in the ointment.)

Fudging

So does anyone have any suggestions as to how to "fudge" non-independent measurements to improve an estimate? For the two most egregious cases I can come up with a fair estimate of how connected the measurements are, but I don't have a feel for how to factor that knowledge in.

Answer 1

You're making a mistake somewhere.

Yes, $P(E|H)$ can be greater than $P(E)$, but $P(E|H)P(H)$ should never be greater than $P(E)$, since

$$ P(E) = P(E|H)P(H) + P(E|\ {\rm not} \ H)P( \ {\rm not} \ H). $$

EDIT: Presumably the "evidence" is the uncertain test, and the "hypothesis" is the result of the "gold standard test". As the other listed probabilities appear to be right (though they're misleading named; I would instead list them as "conditional probability of positive test result given (positive) negative hypothesis), the marginal hypothesis probabilities (which you don't list) must be wrong, and indeed the Bayes multiplier is right, and greater than one. (But it's a bad test where a positive result increases your belief of a negative hypothesis.)

Initially $P(H = t)$ is just the fraction of gold-standard tests that are true, 2845/6134. Similarly, $P(H = f) = 3289 / 6134$.

$$ \begin{align*} P(H = f | E = t ) &= P(E = f | H = t) P(H = f) / P(E = t) \\ &= (1158/3289)*(3289/6134)/(1623/6134) \\ &= 1158/1623 \end{align*} $$

As should be expected, when you know the test returns true, you get out the fraction of true tests that are revealed by the gold standard test to be negative.

You'll have to be careful when trying to iterate Bayes rule. The next "$P(H | E)$" needs to be "$P(H | E_1, E_2)$", because you're interested in the chances after collecting both bits of data. It's not that hard to setup the formulas, but you have to be extremely careful about what your symbols mean. Jaynes recommended always writing probabilities as probabilities conditioned on "background information". Adding this conditioning to the standard formula may help you see how the formula must be used when iterating.

Answer 2

The mistake is in thinking that $P(E)$, the marginal probability, is some sort of "absolute" probability, which never changes. It should really be written $P(E|I)$ to denote that it is still conditional on something. Here's why

If we denote two pieces of evidence $E_1$ and $E_2$, then we have:

$$P(H|E_1 E_2 I)=\frac{P(H|I)P(E_1 E_2|HI)}{P(E_1 E_2|I)}$$

Now we can use the product rule to decompose $P(E_1 E_2|I)=P(E_1|I)P(E_2|E_1 I)$, and similarly, $P(E_1 E_2|HI)=P(E_1|HI)P(E_2|E_1HI)$. We now get

$$P(H|E_1 E_2 I)=\frac{P(H|I)P(E_1|HI)}{P(E_1|I)}\times\frac{P(E_2 | E_1HI)}{P(E_2 | E_1I)}$$

But the first term is simply the posterior after observing $E_1$, denoted by $P(H|E_1I)$. We now have:

$$P(H|E_1 E_2 I)=P(H|E_1I)\frac{P(E_2 | E_1HI)}{P(E_2 | E_1I)}$$

If I was to write $X=E_1I$, then we have $P(H|E_2 X)=P(H|X)\frac{P(E_2 | HX)}{P(E_2 | X)}$. This gives you the usual "posterior to prior" relationship in sequentially digesting data. The posterior after observing $E_1$ is equal to the prior before observing $E_2$

But we also have that the marginal probability (and the condtional probability) now depends on $X=E_1I$, and not just on $I$. The mistake comes from assuming that they are equal - this is an easy thing to do if your notation does not explicitly recognise prior information somewhere.

The "Bayes multiplier", while not less than $1$, is bounded. To see this, expand $P(E|I)$ using the sum rule as follows:

$$P(E|I)=P(EH|I)+P(E\overline{H}|I)\geq P(EH|I)=P(H|I)P(E|HI)$$ $$\implies \frac{1}{P(H|I)}\geq\frac{P(E|HI)}{P(E|I)}$$

And note that this upper bound is just enough to ensure the posterior probability is less than or equal to 1

Answer 3

For the record, I came up with the following "fudge" scheme:

For each "test", assign to each possible outcome a list of "fudge factors" for each outcome of each succeeding test. The "fudge factor" is 1.0 if neutral, less than 1 (eg, 0.8) if the earlier outcome influences the later one positively, and greater than one if the influence is negative. Since most such factors are judged to be 1.0, only the exceptions need to be listed.

If a particular outcome of the earlier test occurs, then its "fudge factor" list is applied to the target outcomes.

To apply the factors, they are "summed" using the following formula:

$$ \log(z)|\log(z)|^a=\log(x)|\log(x)|^a+\log(y)|\log(y)|^a. $$

where x and y are two "fudge factors" (or one is the previous "sum" value, for more than two) and a is a constant (currently 2.0).

If an outcome of a later test has any non-zero "fudge factors" reaching it, the "multiplier" of the affected outcome is raised to the power of this resulting "sum".

This (after playing with the "fudge factors" a bit) has produced a reasonably stable result which seems to correlate well with "reality".