1
$\begingroup$

In order to compute the power of a statistical test ( eg https://en.wikipedia.org/wiki/Power_of_a_test#Example ) , we need to define "what happens" in the non-null hypothesis. Whereas what happens in the null hypothesis is always simple, what happens in the non-null hypothesis requires to define a priori a value for how different we assume the two populations to be (noted by theta in the link).

How should this theta be computed ? In some tests I've seen in the wild, we take as theta the observed difference of means. Is this OK to do or should the theta be defined before the test ?

Improve this question
$\endgroup$
5
  • $\begingroup$ Abstractly, the alternate hypothesis $H_A$ is a set of distributions. The power of a test of $H_0$ against $H_A,$ by definition, is a function from $H_A$ into $[0,1].$ There is no "computation" or "definition" of $\theta:$ it merely stands for an arbitrary element of $H_A.$ An analogy might help: consider the function $f(x)=x^2$ from $\mathbb R$ to $\mathbb R.$ Your question asks, "in order to analyze this function, how do we compute $x$?" $\endgroup$
    – whuber ♦
    Commented Jan 7, 2022 at 19:02
  • $\begingroup$ I know that there is no computation for theta, but how is it usually chosen ? what is considered the best practice. It's tempting to not think too much about it and choose theta as the observed difference of means, but I'm not sure it's OK to do so $\endgroup$
    – lezebulon
    Commented Jan 7, 2022 at 19:04
  • 2
    $\begingroup$ There is no "best practice!" In order to investigate power, you need to compute the false negative rate as a function of $H_A.$ That's it: it's the definition. Are you perhaps trying to ask how one specifies an alternate hypothesis? If so, that's so general that it would be better to explain your particular problem so we can offer specific answers. $\endgroup$
    – whuber ♦
    Commented Jan 7, 2022 at 19:35
  • $\begingroup$ Maybe see this recent Q&A with two approaches to the answer and some worthwhile discussion in Comments. // A 'power and sample size' procedure is most useful if done in the planning stages of an experiment. Often, this requires using past experience with similar kinds of experiments to get reasonable 'guesses' for some necessary quantities. $\endgroup$
    – BruceET
    Commented Jan 7, 2022 at 23:11
  • 2
    $\begingroup$ Power is a function of $\theta$ (as well as $n$ and $\alpha$), as whuber points out. Do a site search for power curve (most of which look at it as a function of $\theta$); there's numerous plots of power curves on site. Further, what happens at the null is not always simple -- not all nulls are point nulls and rejection rates (but not actually power in this situation) can vary within non-atomic nulls. $\endgroup$
    – Glen_b
    Commented Jan 8, 2022 at 3:52

1 Answer 1

Reset to default
2
$\begingroup$

Suggestion: Obtain $\theta$ a priori by answering the question What is the smallest difference in means that you would find important?

For example, if looking at mean mortality under one policy versus another policy, a difference of 1 death per million people per year might not be 'big enough to matter', while a difference of at least 2 deaths per 100,000 per year might be the boundary of 'big enough to care about' and 'not big enough to care about'. These are questions about researcher values, and cannot be derived from your data. They also cannot have standard answers.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Note sure what you mean about "researcher values" ? $\endgroup$
    – lezebulon
    Commented Jan 7, 2022 at 19:59
  • $\begingroup$ @lezebulon Values $\endgroup$
    – Alexis
    Commented Jan 7, 2022 at 20:10
  • 1
    $\begingroup$ Ahh I got it :) $\endgroup$
    – lezebulon
    Commented Jan 7, 2022 at 20:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.