How to understand probability of Necessity (PN) ≥ 100%, as in this example from 'Causal Inference in Statistics a primer'

Question

In the book 'Causal Inference in Statistics A Primer' By Pearl et al. there is an example towards the end, (Ex 4.5.1 page 119) that calculates the probability of necessity PN = 1, and the authors state 'The data provides us with 100% assurance that drug x was in fact responsible for the death of Mr. A'

This is a beginner book on the topic, so I'm no expert but this example is raising a lot of red flags for me. I would appreciate some clarification as to how results makes logical sense. At the moment I'm convinced there are multiple errors being made.

Here are my points of contention:

• Stating an event is 100% responsible for anything does not seem realistic. Shouldn't it at max approach 100%?
• If the example data had a single death in the controlled experiment shifted from do(x') to do(x), the formula yields a PN of 200%. If interpreted similarly to the given example, this would be a "200% assurance that the drug was responsible for the death of Mr. A." This outcome seems meaningless as there can not be a 200% probability of something occurring.
• What justifies assuming the two groups came from similar populations.

To keep my post length down, I'll stop with those questions for now.

The author seems to have intended the result to be counter intuitive, and yet all he did was plug in numbers to the formula and present a counter intuitive result without further explanation. Without justification, I have not been able to reconcile the example and am now doubting the correctness of the equations.

Here is the example:

Example 4.5.1 (Attribution in Legal Setting) A lawsuit is filed against the manufacturer of a drug x. charging that the drug is likely to have caused the death of Mr. A, who took it to relieve back pains. The manufacture claims that experimental data on patients with back pains show conclusively that drug x has only minor effects on death rates. However, the plaintiff argues that the experimental study is of little relevance to this case because it represents average effects on patients in the study, not on patients like Mr A who did not participate in the study. In Particular argues the plaintiff, MR. A is unique in that he used the drug of his own volition, unlike subjects in the experimental study who took the drug to comply with the experimental protocols. To support this argument, the plaintiff furnishes non-experimental data on patients who like Mr A, chose drug x to relieve back paints but were not part of any experiment and who experience lower death rates that those who didn't take the drug [How does one have data on patients who didn't take the drug from a sample of those who chose to take the drug?]. The court must now decide based on both the experimental and non-experimental studies whether it is "more probably than not" that drug x was in fact the cause of Mr A's death.

The experimental data provide the estimates

P(y|do(x)) = 0.016
P(y|do(x')) = 0.014

whereas the non-experimental data provide the estimates

P(y) = 30/2000 = 0.015
P(x,y) = 2/2000 = 0.002
P(y|x) = 2/1000 = 0.002
P(y|x') = 28/2000 = 0.028

Data:

               Experimental     NonExper.
                do(x)   do(x')  x      x'  
Deaths(y)        16     14      2     28
Survivals(y')   984     986     998   972

Assuming that drug can only cause but not prevent death, monotonicity holds and therm 4.5.1 yields:

$PN = \frac{P(y|x) - P(y|x')}{P(y|x)} + \frac{P(y|x') - P(y|do(x'))}{P(x,y)}$

$PN = \frac{0.002 - 0.028}{0.002} + \frac{0.028 - 0.014}{0.001} = -13 + 14 = 1$

We see that while the observation ERR is negative (-13) giving the impression that the drug is actually preventing deaths , the bias-correction term (+14) rectifies this impression and set the probability of necessity (PN) to unity. Moreover, since the lower bound of Eq. 4.30 [Not shown by me] becomes 1, we conclude that PN=1.00 even without assuming monotonicity. Thus the plaintiff was correct, barring sampling errors, the data provide us with 100% assurance that drug x was in fact responsible fro the death of Mr. A.

Please help me understand how a result can be "100% assurance" and how the equation makes sense given that if the example experimental data had been shifted by one as shown below:

               Experimental     NonExper.
                do(x)   do(x')  x      x'  
Deaths(y)        17     13      2     28
Survivals(y')   984     986     998   972

Then the result would have been:

$PN = \frac{0.002 - 0.028}{0.002} + \frac{0.028 - 0.013}{0.001} = -13 + 15 = 2$

Which, in the language of the author, would be a seemingly meaningless 200% assurance.

Thanks for your patience with this long post! Please help me with what I'm not understanding about the interpretation and calculation of PN in this example.

Answer 1

Short answer: The equations are correct. What you wrote is a numerical example that is not compatible with the assumption that the non-experimental and experimental data come from the same population. That is, whenever the data comes from the same population, the frequencies you wrote are impossible. See details below.

Long answer: These are very good questions, let's go over each of your points. But let me start with the last two points first.

If the example data had a single death in the controlled experiment shifted from do(x') to do(x), the formula yields a PN of 200%. If interpreted similarly to the given example, this would be a "200% assurance that the drug was responsible for the death of Mr. A." This outcome seems meaningless as there can not be a 200% probability of something occurring.

First, it is important to recall that here we are assuming we have population data. Practically speaking, you should read every table in the Primer as if you had thousands or hundreds of thousands of subjects, not one or two. So the shift of a "single death" in this context is like a shift of thousands or hundreds of thousands of deaths. Formally, it means that the population value of $P(y_{x'})$ "shifted" from $0.014$ to $0.013$.

That said, now let us suppose that $P(y_{x'})$ was indeed $0.013$ instead of $0.014$, and that the observational frequencies remain the same. What would that imply?

In this case, you have what we call a testable implication. If the experimental data for $P(y_{x'})$ was indeed $0.013$, this means that the experimental data and the observational data cannot be from the same population. Or more generally, either the data is wrong, or the assumption that they come from the same population is wrong. In other words, there is no causal model that could create this set of experimental and observational data.

To see why, use the law of total probability to write:

$$ P(y_{x'}) = P(y_{x'}|x)P(x) + P(y_{x'}|x')P(x') $$

We know $P(x) = P(x') = 1/2$. Also, by consistency, $P(y_{x'}|x') = P(y|x') = 0.028$, thus we have that

$$ P(y_{x'}) = \frac{1}{2} \times 0.028 + \frac{1}{2}\times P(y_{x'}|x) $$

By setting $P(y_{x'}|x)$ to one and then to zero we obtain bounds on $P(y_{x'})$,

$$ 0.014 \leq P(y_{x'}) \leq 0.514 $$

That is, given the observed frequencies of the non-experimental data, the minimum value of $P(y_{x'})$ which is still compatible with that data is 0.014. In your example that value is 0.013, which implies that the non-experimental and experimental data cannot be from the same population.

If you want a nice graphical display of the experimental data which is compatible with a given non-experimental data set, you can check out this new blog post by Scott Mueller and Judea Pearl.

This brings us to your point:

What justifies assuming the two groups came from similar populations.

This is an assumption that must be justified by the researcher in each case. Sometimes this assumption can be falsified by the data, as your numerical example shows. But often it cannot, and it must be defended with contextual knowledge. That's usually the case in causal inference, most assumptions cannot be tested from data alone.

Now let me answer your first question.

Stating an event is 100% responsible for anything does not seem realistic. Shouldn't it at max approach 100%?

Let's start with a simpler model in which, hopefully, little dispute may arise.

Suppose you shoot a healthy person and that person dies. Then I ask you: what is the probability that the person would have been alive, had you not shot them? I believe most people will agree that, for all practical purposes, 100% is a reasonable number. How can you compute this probability? You have a model, in your head, of how things work (e.g., healthy humans don't suddenly drop dead), and the only scenario which is compatible with the observed data is that the shot was responsible for the person's death.

So if you agree with the previous argument, you do agree that the answer 100% is possible and can be realistic, provided we agree the model assumptions are reasonable. Thus any doubts regarding the computed probability itself, correspond to doubts regarding the model assumptions, not the principle by which we derived the answer. The same thing holds for the example above. You need to agree on the model assumptions that: (i) the experimental and non-experimental data come from the same population; (ii) there are no measurement errors and so on. It's fine to cast doubt on the assumptions, and some of them can have testable implications as we have seen above. But once you agree the assumptions are reasonable, you are bound to agree with its logical implications.