# How do you choose the "theta" of power of a test?

In order to compute the power of a statistical test ( eg https://en.wikipedia.org/wiki/Power_of_a_test#Example ) , we need to define "what happens" in the non-null hypothesis. Whereas what happens in the null hypothesis is always simple, what happens in the non-null hypothesis requires to define a priori a value for how different we assume the two populations to be (noted by theta in the link).

How should this theta be computed ? In some tests I've seen in the wild, we take as theta the observed difference of means. Is this OK to do or should the theta be defined before the test ?

• Abstractly, the alternate hypothesis $H_A$ is a set of distributions. The power of a test of $H_0$ against $H_A,$ by definition, is a function from $H_A$ into $[0,1].$ There is no "computation" or "definition" of $\theta:$ it merely stands for an arbitrary element of $H_A.$ An analogy might help: consider the function $f(x)=x^2$ from $\mathbb R$ to $\mathbb R.$ Your question asks, "in order to analyze this function, how do we compute $x$?"
– whuber
Jan 7, 2022 at 19:02
• I know that there is no computation for theta, but how is it usually chosen ? what is considered the best practice. It's tempting to not think too much about it and choose theta as the observed difference of means, but I'm not sure it's OK to do so Jan 7, 2022 at 19:04
• There is no "best practice!" In order to investigate power, you need to compute the false negative rate as a function of $H_A.$ That's it: it's the definition. Are you perhaps trying to ask how one specifies an alternate hypothesis? If so, that's so general that it would be better to explain your particular problem so we can offer specific answers.
– whuber
Jan 7, 2022 at 19:35
• Maybe see this recent Q&A with two approaches to the answer and some worthwhile discussion in Comments. // A 'power and sample size' procedure is most useful if done in the planning stages of an experiment. Often, this requires using past experience with similar kinds of experiments to get reasonable 'guesses' for some necessary quantities. Jan 7, 2022 at 23:11
• Power is a function of $\theta$ (as well as $n$ and $\alpha$), as whuber points out. Do a site search for power curve (most of which look at it as a function of $\theta$); there's numerous plots of power curves on site. Further, what happens at the null is not always simple -- not all nulls are point nulls and rejection rates (but not actually power in this situation) can vary within non-atomic nulls. Jan 8, 2022 at 3:52

Suggestion: Obtain $$\theta$$ a priori by answering the question What is the smallest difference in means that you would find important?