Let's take a simple one-sided hypothesis test (say the two sample t-test). The confidence level we want (probability of rejecting the null when it is true) is defined as $\alpha$ - usually set to 5%. Now, I want to compare the two sample t-test with a different hypothesis test (say I use the normal distribution instead of the t-distribution in the test) and see how much better it is. One way to decide which test is better is to get the $\beta$ (defined as the probability of failing to reject the null hypothesis when it is false - meaning the alternate hypothesis is true). Then, $1-\beta$ becomes the power of the test corresponding to the $\alpha$ we choose. The test with higher power is better.

To calculate $\beta$, we need to basically compute the CDF of a distribution under the alternate hypothesis. The problem is, the alternate hypothesis is generally defined as the treatment group have a "larger" mean (in the context of the two sample t-test) than the control group. This would become a whole family of distributions (how much larger).

Now, which member of this family do we choose to calculate $\beta$?

One option is to define an effect size we want to capture (say we're interested in a difference of mean: 10), get the distribution under this difference assumption and get the $\beta$ using it.

Another option is to integrate over all positive effect sizes ($0$ to $\infty$). This is obviously going to be significantly harder than the first approach.

Is there any other approach we can use to compare the powers? And between the two approaches mentioned above, will the second one be substantially better than the first one (for which we have to choose an "arbitrary" effect size)?

  • $\begingroup$ For t-test vs z-test, first thing to consider is how to keep the alpha level at 0.05. $\endgroup$ – user158565 May 3 '19 at 2:40
  • 1
    $\begingroup$ 1. Rather than choose an effect size (say, $\delta$), or integrate over effects sizes (for which you'd need to choose some distribution of effect size), you could also calculate a power surface as a function of both $\alpha$ and $\delta$. Another option would be to choose some specific power at a given $\alpha$ (like social science's commonly chosen 80% power at $\alpha=0.05$ at some specific effect size and sample size) and then compare power as you vary $\alpha$ from that. But none are better in every sense; you'd need to decide what you want to examine, or clarify what you want to know. $\endgroup$ – Glen_b May 3 '19 at 4:48
  • 1
    $\begingroup$ 2. similarly, in the last paragraph, when you say "better" we'd need to know the sense in which you intend it (better at doing what?) $\endgroup$ – Glen_b May 3 '19 at 4:48
  • $\begingroup$ I have two different hypothesis tests. Let's say (not that it matters) one of them is the two sample t-test and the other is the same test, using a Gaussian instead of t-distribution. I want to know which of these is a better hypothesis test and how much better it is than the other. I don't know anything about the effect size that will be used in the experiment. To answer this question, I want to plot $\beta$ as a function of $\alpha$ for both the tests and compare the plots. Which methodology should I use to calculate $\beta$ for this purpose? $\endgroup$ – ryu576 May 3 '19 at 4:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.