# What's the probability that this is true under the null hypothesis?

## Scenario

Suppose that there is the population of Raspberry Pies (RPs).

Suppose that you sample 5 of those RPs, and after 1 month, you find that 3 of them have some problem (faulty chip). So the sample probability that RPs have a problem is 3/5.

Then suppose that you find a magical spray can that says "if you spray this on RPs, they shall be purified from all chip illnesses".

Suppose that you sample another 5 RPs, except that you immediately spray them with that magical purification spray.

Suppose that you wait 1 month, and: all of the 5 recently sampled RPs have absolutely no faulty chips in a month (which is an improvement over the previously sampled RPs that got the faulty chip in 1 month).

## Hypothesises

• H0 - the magical spray is a scam. It makes no difference, and the observed enhancement is due to sheer dumb luck.

## Question

• Suppose that H0 is true, what is the probability to observe the above scenario?

• Suppose that I tell you that all quantities of RPs that I have samples, were 1000 more instead of just 5. What difference would this information make to your answer with respect to the first question?

• Is my attempt below exactly what's called Fisher's exact test?

• Is my observation OBSERVATION 1 exactly likelihood maximization?

## My attempt

If we suppose that H0 is the case, then it means that having all the 5 RPs not die in a month after getting magic-sprayed is due to sheer dumb luck.

That means that, the probability that a RP fails after a month is independent of whether it got magic-sprayed. If I abuse notation, that means: $\Pr(\text{RP dies after 1 month}) = \Pr(\text{RP dies after 1 month}|\text{got_magic_sprayed})$.

REMARK 1: Therefore it means that, if H0 is true, then the probability of observing a dead RP is just $\Pr(\text{RP dies after 1 month}) = 3/5$.

GUESS 1: I guess H0 must also imply that the death of any RPs is independent of the state other RPs. (correct me please).

Using REMARK 1 and GUESS 1, it seems that (if H0 is true) sampling any 5 RPs with 3 dead RPs after 1 month is:

• Plus ... all combinations of 3 deads and 2 oks.

We have ${5 \choose 3}$ many such combinations, with each combination having the probability to occur (under H0) $(3/5)^3 (1-3/5)^2$. I.e. ${5 \choose 3} (3/5)^3 (1-3/5)^2 = 0.3456$.

Now, applying the same logic, what is the probability of sampling 5 RPs such that ALL of them die after 1 month assuming H0 holds? Here is this: ${5 \choose 5} (3/5)^5 (1-3/5)^0 = 0.216$.

ANSWER 1: Now, same but zero deads, all OK: ${5 \choose 0} (3/5)^0 (1-3/5)^5 = 0.01024$. Oh my godness, did you see that? Since $p \approx 0.01$ is too tiny (less than than $0.05$, I reject H0, which necessarily implies that the magic spray can is not a scam!

OBSERVATION 1: assuming that H0 is true, the measured probability is maximum only if number of dead RPs is 3, and ok RPs is 2.

PROBLEM 1: if instead of sampling 5, I sample a 1000, then the probability of observing 3:5 dead to OK RPs (under H0) is: ${1000 \choose 600} (3/5)^{600} (1-3/5)^{400} = 0.026$....!!! This -too- is statistically significant? But how can it be, it's just the null hypothesis itself, except for observing a larger sample.

GUESS 2: I guess we must somehow normalize the $p$ value of the observation against the maximum likelihood that is $p=0.026$. Right?

• In statistical inference, you test your data under the hypothesis that null is true. The probabilities that you calculate (e.g. p-values) are not of whether the null is true. They're probabilities of consistency of the data with true null. Mar 19 '16 at 16:18
• Thank you. Updated question. Is it OK? Mar 19 '16 at 16:28
• Now that you are focused on the probability of the data rather than of the hypothesis, this is a standard problem in analysis of contingency tables. This seems to be a self-study type of question, so please read the information on this site about self-study, indicate what you've done so far and where you are stuck. Also, the present version of the question still has the fatal experimental design flaw noted in my answer. Even superb statistical analysis can't rescue faulty experimental design.
– EdM
Mar 19 '16 at 16:35
• Added stuff for compliance with respect to self-study questions. Mar 20 '16 at 5:57