I wanted to dig a bit deeper into hypothesis testing and fresh up my conceptual understanding from undergraduate courses. The typical way of how hypothesis testing is teached is to calculate the Z-score and then compare the associated P-value with a pre-determined cut-off level of alpha.
Let's create a hypothetical example. I've found a new drug for which I claim it raises the level of blood value X. We know that the average blood level of X in the control group is X = 10 with a standard deviation of 1.
I've ingest the drug to my friend now and measure his blood level afterwards. E voila, his blood level is 12. Given the formula above, I would arrive at a Z-Score of 2 which is far below the normally used cut-off level of 0.05
Coming to my question
How can I conclude validity from this result? E.g it could have been that I have given the drug to a group of 100 people and observed that the average result was the same as in the control group. Similarly, I could have found my friend in the group of 100 people with his extreme value of 12. I could repeat the hypothesis test only with im and arrive at a statistically meaningful result. Ta Da!
Could someone verify if my understanding is flawed here? Thank you very much!