I wanted to dig a bit deeper into hypothesis testing and fresh up my conceptual understanding from undergraduate courses. The typical way of how hypothesis testing is teached is to calculate the Z-score and then compare the associated P-value with a pre-determined cut-off level of alpha.

enter image description here

Let's create a hypothetical example. I've found a new drug for which I claim it raises the level of blood value X. We know that the average blood level of X in the control group is X = 10 with a standard deviation of 1.

I've ingest the drug to my friend now and measure his blood level afterwards. E voila, his blood level is 12. Given the formula above, I would arrive at a Z-Score of 2 which is far below the normally used cut-off level of 0.05

Coming to my question

How can I conclude validity from this result? E.g it could have been that I have given the drug to a group of 100 people and observed that the average result was the same as in the control group. Similarly, I could have found my friend in the group of 100 people with his extreme value of 12. I could repeat the hypothesis test only with im and arrive at a statistically meaningful result. Ta Da!

Could someone verify if my understanding is flawed here? Thank you very much!


I think your understanding is basically sound here. If you conduct the study as you describe, with only one patient taking the drug, and measure $x = 12$, that would constitute a significant result, so the conclusion would be that the drug does work.

While this is formally correct, I suppose most of us would be suspicious about this result, in particular in view of the extremely small sample size. And indeed, the conclusion may still be wrong, not only because of the obvious fact that $\alpha = 0.05$ is still more than zero, but also because the entire statistical argument here rests on some very strong assumptions, e.g.

  1. The blood level measurements are normally distributed under the null hypothesis (with mean $\mu$ and standard deviation $\sigma$).
  2. Your friend is randomly sampled from the population of all people who might take the drug.
  3. You know the exact value of $\mu$.
  4. You know the exact value of $\sigma$.

Especially assumptions 3 and 4 are so unrealistic that they are usually not made in practice. So you usually have to estimate $\mu$ and $\sigma$ too, and the simple Z-test is no longer applicable.

Regarding your penultimate paragraph, if you have done a study with 100 people, then you should of course include all of them in the calculations (except for certain pre-specified exclusion criteria). Basing the test on only one (or several) person(s) with high $x$ would be statistical malpractice ("cherry-picking"), because assumption 2 would no longer hold.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.