4
$\begingroup$

This problem was taken from a free online Computational Neuroscience class {1}.

Suppose we are diagnosing a very rare illness, which happens only once in 100 million people on average. Luckily, we have a test for this illness but it is not perfectly accurate. If somebody has the disease, it will report positive 99% of the time. If somebody does not have the disease, it will report positive 2% of the time.

Suppose a patient walks in and tests positive for the disease.

Using the maximum likelihood (ML) criterion, would we diagnose them positive? What if we used the maximum a posteriori (MAP) criterion?

I'm curious about the actual maths behind the decision outcomes. The correct answers are Yes and No, respectively, but I'm unsure of how to analytically derive those solutions. I would love to see some of the work texted up.


References:

$\endgroup$
  • $\begingroup$ Just a side note attacking the question posed by the class: the ML and MAP answer should be identical. What the questioner is considering to be the prior (proportion of population with disease) can easily just be joined directly into the likelihood function itself. There is no rule banning hierarchical models from maximum likelihood estimation. $\endgroup$ – Cliff AB Jan 2 '17 at 23:38
  • $\begingroup$ This question was asked here previously. Were you the OP. If so what was the disposition? $\endgroup$ – Michael R. Chernick Jan 3 '17 at 23:07
6
$\begingroup$

If we let $d=\mathrm{disease}$, $\sim\!d=\text{not disease}$, and similarly for $t=\mathrm{test}$, then I believe the two approaches are comparing the conditional probabilities: \begin{array}{} \text{MLE}:& \text{positive} &\implies \Pr\big[\,t\mid\,d\,\big]>\Pr\big[\,\phantom{\sim}\,\,\,t\mid\,\sim\!d\,\big] \\ \text{MAP}:& \text{positive} &\implies \Pr\big[\,d\mid\,t\,\big]>\Pr\big[\,\sim\!d\mid\phantom{\sim}t\,\,\big] \\ \end{array} The problem statement gives you the prior probability, $\Pr\!\big[\,d\,\big]$ (a.k.a. base rate), along with the true and false positive rates, $\Pr\!\big[\,t\mid\,d\,\big]$ and $\Pr\!\big[\,t\mid\sim\!d\,\big]$.

To compute the MAP diagnosis you need the posterior probabilities, which can be computed using Bayes' theorem* and the law of total probability. (*The "Examples" section here includes several problems directly analogous to yours.)

Finally, note that the example serves as a demonstration of the dangers of "base rate neglect". This is because the MLE diagnosis only considers the likelihood ratio, while the MAP diagnosis also incorporates the base rate.


Note: As pointed out by Mark Stone, in the real world self selection bias should be considered (i.e., the base rate among patients seeking a test will generally differ from the population base rate). The "textbook" solution is more relevant to screening.

| cite | improve this answer | |
$\endgroup$
4
$\begingroup$

I strongly disagree in the real world with the answer provided in the question and the answers which have already been provided by @GeoMatt22 and @user99889 (I don't blame them as they are showing how to get the answer in the question). In the fake world of a class problem by a professor who is not being realistic, this is probably the correct answer in the sense of that's what the professor wants the student to give as the answer. In the real world, it can be off by orders of magnitude. The answer provided in the question exhibits a very common fallacy among people who are sophisticated enough to try to use Bayes' theorem, but not sophisticated enough to think things through properly.

An implicit, but possibly very invalid assumption is being made that the population of patients who walk in and get tested for the disease has the same distribution of the very rare illness occurrence as the population of "all" people for whom the occurrence is 1 in 100 million. It may be that only patients exhibiting symptoms, or having a family history, or having already received certain other test results are the ones who are getting tested for the very rare illness. Or at least that such people are more likely than people in the whole population, to be the ones getting tested. Now the prior distribution of having the very rare illness is not 1 in 100 million, it might be orders of magnitude higher. Then of course there's the matter of dependence ("correlation") between test results among different tests given a sequential sequence of different tests.

| cite | improve this answer | |
$\endgroup$
  • 4
    $\begingroup$ (+1) Yes, in the real world self selection bias should be considered. The "textbook" solution is I believe more relevant to screening. $\endgroup$ – GeoMatt22 Jan 2 '17 at 16:26
  • $\begingroup$ @user99889 But that's not what the online course content creator was doing. $\endgroup$ – Mark L. Stone Jan 2 '17 at 23:17
  • $\begingroup$ GeoMatt22 is right for screening false positives can be a real problem. That is why screening tests are often repeated or if the subject tests positive on the first test a more sensitive test is provided to confirm the diagnoses. $\endgroup$ – Michael R. Chernick Jan 3 '17 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.