1
$\begingroup$

If I have an experiment documenting the recovery of 1,000 patients who were given access to a medication.

Results:

  • MEN taking the drug. 2 out of 2 recover (100% recovery).
  • MEN not taking the drug. 400 out of 498 recover (80.3% recovery).
  • WOMEN taking the drug. 30 out of 498 recover (6.02% recovery)
  • WOMEN not taking the drug. 0 out of 2 recover (0%recovery).

People taking drug: 32 out of 500 recover (6.4% recovery). People not taking the drug: 400 out of 500 recover (80% recovery).

Can I conclude the next statement?

Given:

  • 100% (drug) vs 80.3% (no drug) recovery for men and
  • 6.02% (drug) vs 0% (no drug) recovery for women,

Therefore: the drug is indisputably helpful.

Is there any flaw in my logic?

(data is fake, but my question is about my conclusion, whether the conclusion is plausible given the data…)

EDIT:

My example was slightly modified from Pearl's book: enter image description here

I know I exagerated the sample size... but he is still comparing samples, which is more than 3 times larger!! (270 men no drug, to 80 women no drug), how is he comparing them as equal? and besides, how he arrives to such huge 'indisputably' claim without even doing any statistical test?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ I see two issues, the second of which may not be an issue in your real data. 1) What is your hypothesis, that the drug is helpful? Helpful to men? Helpful to women? Helpful overall? 2) You have a lot of women taking the drug but very few men...why? $\endgroup$ – Dave Mar 5 '20 at 2:55
  • $\begingroup$ Well, there is no actual hypothesis… The example is slightly adapted from the first pages of the book “Causal Inference in Statistics: A Primer” from Pearl. When I first read it,it made sense, but now that I go through it, it seems completely naïve. I do not know why we have way more women taking the drug than men, can we even compare it like that? I do think the sample size does matter! $\endgroup$ – Chicago1988 Mar 6 '20 at 1:27
3
$\begingroup$

To conclude that the drug is "indisputably helpful" you should use a statistical test, like a hypothesis test with Kaplan Meier estimations of survival functions. Or you could use a regression model like Cox Proportional Hazards to quantify the effect of the drug.

Improve this answer
$\endgroup$
1
  • $\begingroup$ The example is slightly adapted from the first pages of the book “Causal Inference in Statistics: A Primer” from Pearl. How come is he able to arrive to such conclusion without any sort of statistical test?? Is it possible? $\endgroup$ – Chicago1988 Mar 6 '20 at 1:27
2
+50
$\begingroup$

There is a journal article titled "Good for women, good for men, bad for people: Simpson's paradox and the importance of sex-specific analysis in observational studies." (Journal of Women's Health & Gender-based Medicine, 01 Nov 2001, 10(9):867-872 DOI: 10.1089/152460901753285769 PMID: 11747681) and yes: Things can be helpfull for people of each gender yet appear to be disadvantageous if looked at without recognizing gender.

However, there is still a flaw in your logic when you consider 100% (2 out of 2) to be superior to 80.3% (400 out of 498). The 95% confidence interval for the true share of 2 out of 2 ranges from 16% to 100%

> binom.test(2,2)$conf.int
[1] 0.1581139 1.0000000
attr(,"conf.level")
[1] 0.95

The 95% Ci for 400 out of 498 ranges from 77% to 84%. Thus, 2 out of 2 is not proof of a higher share then 400 out of 498:

> fisher.test(matrix(c(2,0,400,98),nrow=2))$p.value
[1] 1

The flaw in your logic is overestimating the value of small samples.

Improve this answer
$\endgroup$
2
$\begingroup$

Therefore: the drug is indisputably helpful.

We can't conclude that at all.

In general, you should come up with a statistical test first, run the experiment, feed the results into the test, and only then draw your conclusion. Otherwise, you are in danger of letting your data suggest a hypothesis which inevitably will be "surprising", perhaps even leading to daft conclusions like this.

Absent of a hypothesis, it's OK to look for weird features in your results, but only to use those to inform the design of a proper statistical test on fresh data. (Well, up to a point: if the drug-takers all burst into flames upon consuming their pills, you could quickly draw a conclusion, albeit qualitative rather than quantitative.)

As it happens, the results you quoted are unsurprising with respect to the effect of the drug. In part, this is due to bad experiment design: some of the classes are really small (just 2 people), so you can shrug-off just about any outcome as a fluke. Consider: given that 402 out of 500 men recovered, the probability that (0,1,2) drug-takers recovered is (0.038,0.318,0.646) respectively -- even the least-likely case isn't super-surprising. In fact, your data are the most likely outcomes for the "drug has no effect" hypothesis, conditional upon the observed totals.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ My example is slightly adapted from the first pages of the book “Causal Inference in Statistics: A Primer” from Pearl. How come is he able to arrive to such conclusion without any sort of statistical test? $\endgroup$ – Chicago1988 Mar 9 '20 at 19:21
  • $\begingroup$ What conclusion, precisely, does the author reach? And from exactly what data? $\endgroup$ – Creosote Mar 9 '20 at 22:42
0
$\begingroup$

I think some of this depends on how the data were obtained. For instance, if these people were randomized to the drug, then conclusions about efficacy are easier to make. But, if this was just found data (or data we could get because it was easy), then we have to be a little more careful. I'll assume that people were randomized to either drug or no drug. Next, I'll assume that "helpful" means that the probability of recovery for those who take the drug is larger than those who do not take the drug.

What you present here is stratified data. The stratifying factor is Male/Female, so let's look at the 2x2 table for each of the strata. For men...

$$ \begin{matrix} & Drug & No Drug \\ \mbox{Recover} & 2 & 400 \\ \mbox{Don't Recover} & 0 & 98 \end{matrix} $$

and for women...

$$ \begin{matrix} & Drug & No Drug \\ \mbox{Recover} & 30 & 0 \\ \mbox{Don't Recover} & 468 & 2 \end{matrix} $$

If you only seek to assess if the drug is "helpful", consider the combined table.

$$ \begin{matrix} & Drug & No Drug \\ \mbox{Recover} & 32 & 400 \\ \mbox{Don't Recover} & 468 & 100 \end{matrix} $$

Using a binomial test, we find that the confidence interval for the difference in probability of recovery between drug and no drug groups spans (-0.77, -0.69); in other words, the data is not consistent with the drug being helpful. IN fact, it looks like the drug is actually causing people NOT to recover (again, I can jump to causal assumptions because I assumed randomization. This, in general, can not be done).

Now, this could be confounded by gender. Way way more women were given the drug as compared to men. If the effect depends on the strata, that could be biasing our pooled result. Let's use a modified poisson regression with robust covariance estimation to jointly analyze the strata. I'll get to that at a later time...

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.