In his 1967 paper “A Theorem for Prediction” (Studies in Intelligence 11(4)), Jack Zlotnick discusses how some CIA experiments lead to a “modification” of Bayes' theorem, to account for unreliable evidence. He uses prior and posterior odds $\mathrm{P}$ and $\mathrm{R}$ and a likelihood ratio $\mathrm{L}$ for the original statement of the theorem, $\mathrm{R} = \mathrm{PL}$, and then introduces a reliability rating $r$, basically the probability that the observation yielding $\mathrm{L}$ is true. As he says:

Call this probability of correct reporting the reliability rating. A 30 percent reliability rating would mean that 30 percent of the reports with such a rating are true […] and the other 70 percent are false.

He then modifies the equation to $$ \mathrm{R} = \mathrm{PL}^r\,. $$ This would seem to make sense (quite obviously so for the extreme cases of $r=0$ or $r=1$), but I'm still a bit uncertain about the derivation of this formula. I'm also a bit confused by Zlotnick's discussion of the term “false”:

False reports are of two kinds. One is bereft of any corresponding fact, the utter fabrication for example. Such a report would be the assistant's announcement of a red poker chip when he had actually picked nothing at all out of the box. […] The second kind of false report is one which deliberately or innocently confuses one event with another, for example the assistant's announcement of a red chip when it was in fact blue. For reports estimated to have a probability of being false in this sense, the required modification of the equation becomes perhaps too involved to explain in a non-mathematical journal, but the mathematics is not really difficult.

The first kind of false report is the one leading to the modified formula (while the second kind isn't discussed any further).

Beyond not immediately seeing how he derives his formula, I'm also a bit confused about his discussion of true and false reports. If $r=.3$ means a 30% probability that the report is true and a 70% probability that it is false (as he says), why does the mechanism behind this matter? (Is it, perhaps, that the event is not random, or not independently so?)

This latter point is really of less importance (although I would have liked to see the “too involved” formula he's talking about); my main question is whether this formula is common knowledge, or if someone could give a solid derivation (treating $r$ as a probability)? I've tried looking around for more material, but just keep running into CIA papers from the sixties (with no further explication).

  • $\begingroup$ If the whole thing is viewed additively, with $\ell = \ln\mathrm{L}$ being the log-odds ratio, then I guess $r\ell$ would be the expected value for the weight of evidence from the unreliable source, if treated as a random variable (which is either $\ell$ or 1, meaning the source is simply useless with a probability of $1-r$). A colleague drew up some reasoning based on the idea of a series of independent pieces of information, where we only observed a proportion $r$, leading to the factor $\mathrm{L}^r$. I'd still like a more definitive answer, though… $\endgroup$ – Magnus Lie Hetland Jan 5 '13 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.