In a double-blind study, when are deviations from the control group considered statistically significant? And is this related to the number of samples?
I realise that every experiment is different and statistical significance should depend on the deviations in measurements and the size of the sample group, but I'm hoping there is an intuitive rule-of-thumb formula that can "flag" an event as interesting.
I have a light stats background in engineering, but am by no means a stats guru. A worked example would be much appreciated to help me understand and apply it to every-day things.
Update with example: OK, here's a (not so simple) thought experiment of what I mean. Suppose I want to measure the toxicity of additives in a village's water supply by comparing mortality rates over time. Given the village's population, natality and mortality rates over several years and the date upon which an additive was introduced into the water supply (disregard quantity), when would a rise in the mortality rate become interesting?
Intuitively, if the mortality rate remains between 0.95% and 1.25% for 10 years, and suddenly spikes to 2.00%, then surely this would be an interesting event if an additive was added that year (assume short-term toxic effects). Obviously there could be other explanations for the rise, but let's focus on statistical significance. Now, how about if it rises to 1.40%? Is that statistically significant? Where do you draw the line?
I'm starting to get the feeling that I need to "choose" a critical region, which feel less authoritative. Can a Gaussian distribution guide me on this? What other information do I need to determine statistical significance?