Suppose I want to test if drug A is effective to reduce the levels of some disease. From previous studies, the drug is estimated to reduce the disease by 0.5 units (mean difference) with a standard deviation of 1.195 units. With a level of significance of 5% and power of 80%, I determine that the sample size is 180 (90 for the placebo group and 90 for the test group). Some months later, I receive the data results after giving the drug to the test group. Should I make a hypothesis test to make conclusions about the drug? Imagine the group with the drug reduced the disease level by 0.4 (<0.5): is the drug not valid? Should I make a hypothesis where the null hypothesis says the mean between both groups is equal and the alternative is that the mean in the test group is lower? Is this hypothesis test independent of the parameters needed to calculate the sample size? Can I make this hypothesis test with a level of significance of 2.5% even though I used 5% for the sample size calculation? I'm pretty confused.

EDIT: Also do I really need to use power to calculate the sample size? Cant I just determine the sample size for two independent samples (continuous outcome) with the formula $n_i=(\frac{z \delta}{E})^2$ where E is the margin error. This formula can be found in some references. After this, I get the size for the groups $n_1$ and $n_2$. But even in this case, can I make a hypothesis test afterward to compare the mean difference and can I use a different confidence level that I used in the sample size determination?


2 Answers 2


Books have been written to answer your questions, as your questions cover a wide statistical ground. Your last question is the easiest one to answer. Basing a sample size on achieving a given precision (margin of error or half width of a confidence interval, etc.) is extremely reasonable and is often recommended over using traditional frequentist statistics power calculations. This also relates to one of your earlier questions. You immediately conceptualized the analysis in terms of a hypothesis test, but that is not the only way to go. In frequentist statistics you could stop with a confidence interval and with Bayesian statistics a posterior distribution.

Next comes the subtle part related to your first questions. You envision the data as arriving "all at once" and you have taken the typical approach of doing a sample size calculation to find out how long to wait before analyzing the data (until you expect the power to cross an arbitrary threshold). Sample size calculations are needed in some cases if you have a fixed budget, and they are needed when doing sequential frequentist analysis because of it's need to "spend $\alpha$" which is the type I assertion probability. If using Bayesian or likelihoodist inference, the sample size is not an intrinsic part of the design and you can experiment until you've learned enough and analyze the data as frequently as desired.

Sequential learning without a firm sample size has many advantages. One of the biggest advantages is that you may be at what you had hoped was the final sample size but you see either more variability in $Y$ than you had planned for, or you see an equivocal result about the treatment effect. You decide it is worth randomizing a few more subjects to see if you can obtain a more definitive result. Contrast that with the rigid design you (and so many others) use in which a very common outcome is something like p=0.1 at the target sample size and a large margin of error, with the conclusion being "we don't know anything now that we didn't know before the experiment was done", i.e., we just know we spent the time and money.

There are at least three more fundamental points that arise from your questions. First, you don't use effects observed in previous studies in power calculations. Power is computed based on an effect you don't want to miss, not on an effect that someone else observed. Second, in your "imagine the drug reduced by 0.4" question, you have to note that the 0.4 is a point estimate with a lot of uncertainty (non-zero margin of error). You can say that the drug on net worked better for subjects in your sample but you can't say the drug worked better in general. Finally you ask about switching to a one-sided test if your (noisy) point estimate is in the favorable direction. It is better to think of that pre-data when formulating the design. But for $\alpha=0.05$ you can think of this as validly allowing for two one-sided $\alpha=0.025$ tests. Again note that an uncertainty interval rather than hypothesis testing would be helpful. Also it is very important to note that you are taking for granted that the traditional frequentist statistical paradigm is the one to use. If you were using Bayesian posterior probability distribution (one that can be updated each time a new observation is collected), most of the probabilities you compute are already directional. For example you can compute the probability the drug is working in the right direction or the probability that the drug is working better than a 0.2 effect. With Bayes there is no multiplicity correction because you might also be asking whether there is evidence that the drug works in the wrong direction.


Should I make a hypothesis test to make conclusions about the drug?

Hypotheses for testing (and other desired inferences) should generally be formulated in the initial planning stages of the experiment, before you see the data. If you formulate hypotheses after you look at the data, this is a form of exploratory data analysis, and it generally means that you cannot validly test those hypotheses without confirmatory bias. However, all is not lost. There are some obvious inferences that would apply in this type of experiment, and so it would not be problematic to get these after seeing your data. The most obvious thing to make an inference about in this case would be the difference between the two drugs on the disease in question. This can be measured by taking a confidence interval for the (population) mean difference, or you could perform a two-sided test of means. (One-sided tests formulated after looking at the data are seriously biased --- do not use them.)

Is this hypothesis test independent of the parameters needed to calculate the sample size? Can I make this hypothesis test with a level of significance of 2.5% even though I used 5% for the sample size calculation?

As a general rule, your inferences (hypothesis tests, confidence intervals, etc.) depend only on the observed data, and are not dependent on the prior quantities you used to determine the sample size for your experiment. So yes, your hypothesis test (or other inferences) should be independent of any parameters you used to compute the sample size. You can choose any significance level you want for your hypothesis test so long as it is formulated independently of the data ---i.e., do not cheat by tweatking the significance level to get a test outcome you prefer. Hence, you can use a different significance level for your testing than the alpha-level you used for the sample size calculation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.