What's the probability that this is true under the null hypothesis?

Question

Scenario

Suppose that there is the population of Raspberry Pies (RPs).

Suppose that you sample 5 of those RPs, and after 1 month, you find that 3 of them have some problem (faulty chip). So the sample probability that RPs have a problem is 3/5.

Then suppose that you find a magical spray can that says "if you spray this on RPs, they shall be purified from all chip illnesses".

Suppose that you sample another 5 RPs, except that you immediately spray them with that magical purification spray.

Suppose that you wait 1 month, and: all of the 5 recently sampled RPs have absolutely no faulty chips in a month (which is an improvement over the previously sampled RPs that got the faulty chip in 1 month).

Hypothesises

• H0 - the magical spray is a scam. It makes no difference, and the observed enhancement is due to sheer dumb luck.

Question

• Suppose that H0 is true, what is the probability to observe the above scenario?

• Suppose that I tell you that all quantities of RPs that I have samples, were 1000 more instead of just 5. What difference would this information make to your answer with respect to the first question?

• Is my attempt below exactly what's called Fisher's exact test?

• Is my observation OBSERVATION 1 exactly likelihood maximization?

My attempt

If we suppose that H0 is the case, then it means that having all the 5 RPs not die in a month after getting magic-sprayed is due to sheer dumb luck.

That means that, the probability that a RP fails after a month is independent of whether it got magic-sprayed. If I abuse notation, that means: $\Pr(\text{RP dies after 1 month}) = \Pr(\text{RP dies after 1 month}|\text{got_magic_sprayed})$.

REMARK 1: Therefore it means that, if H0 is true, then the probability of observing a dead RP is just $\Pr(\text{RP dies after 1 month}) = 3/5$.

GUESS 1: I guess H0 must also imply that the death of any RPs is independent of the state other RPs. (correct me please).

Using REMARK 1 and GUESS 1, it seems that (if H0 is true) sampling any 5 RPs with 3 dead RPs after 1 month is:

• Probability of this: dead, dead, dead, ok, ok.
• Plus that of this: dead, dead, ok ,dead, ok.
• Plus ... all combinations of 3 deads and 2 oks.

We have ${5 \choose 3}$ many such combinations, with each combination having the probability to occur (under H0) $(3/5)^3 (1-3/5)^2$. I.e. ${5 \choose 3} (3/5)^3 (1-3/5)^2 = 0.3456$.

Now, applying the same logic, what is the probability of sampling 5 RPs such that ALL of them die after 1 month assuming H0 holds? Here is this: ${5 \choose 5} (3/5)^5 (1-3/5)^0 = 0.216$.

ANSWER 1: Now, same but zero deads, all OK: ${5 \choose 0} (3/5)^0 (1-3/5)^5 = 0.01024$. Oh my godness, did you see that? Since $p \approx 0.01$ is too tiny (less than than $0.05$, I reject H0, which necessarily implies that the magic spray can is not a scam!

OBSERVATION 1: assuming that H0 is true, the measured probability is maximum only if number of dead RPs is 3, and ok RPs is 2.

PROBLEM 1: if instead of sampling 5, I sample a 1000, then the probability of observing 3:5 dead to OK RPs (under H0) is: ${1000 \choose 600} (3/5)^{600} (1-3/5)^{400} = 0.026$....!!! This -too- is statistically significant? But how can it be, it's just the null hypothesis itself, except for observing a larger sample.

GUESS 2: I guess we must somehow normalize the $p$ value of the observation against the maximum likelihood that is $p=0.026$. Right?

Answer 1

The way you originally posed the question in terms of the probability of the hypothesis, there is no answer in classical (frequentist) statistics, and you need to provide further information for an answer in the Bayesian framework.

In frequentist analysis the hypothesis is either true or false. There is no "probability" of the hypothesis, as this answer to one of the earliest questions on this site makes quite clear. Frequentist analysis examines the probability of obtaining a certain set of data given that a particular hypothesis is true.

Bayesian analysis considers the probability of an hypothesis, but it begins with a prior assumption about that probability, which you have not provided. Data are then used to update that probability. As you can imagine, the manufacturer of the magical spray would be quite likely to challenge any prior probability assumption that you make in this case.

If you intended to ask about probability of data rather than of hypotheses, this is a simple problem in binomial statistics and contingency tables. See this question for discussion of statistical tests, this Wikipedia page for background, and the Vassar site as a resource for doing computations (look for 2x2 contingency table tests).

As you posed the situation, with one set of results obtained before the other, without direct comparison, you may have a problem with experimental design. My first thought was that the first batch of RPs was faulty, and those receiving the magical spray came from a better batch.