4
$\begingroup$

Assuming a test where p > alpha and n is large enough for power > 95% at effect size d, what is the exact interpretation of the test regarding the relationship between the observed data, the real effect, and power for d?

Some details: if my p is larger than my alpha, that means the observed data are not surprising from the perspective of a nil-null hypothesis. But according to Cohen (1990, p. 1309), the failed test, in combination with an estimate of power given a d, also allows me to estimate something similar to, but not actually the following statement: based on my sample, the real population effect is likely (where "likely" is somehow related to my 95% power) as close or closer to no effect than the d I have calculated my power for (not the d I have measured). However, I am not aware of a precise definition, and this statement is definitely false since it interprets the data from the perspective of p(H|D) and not p(D|H)…

I am looking for a statement comparable to “given a p value below alpha, assuming a zero effect, observing data as or more extreme than the evaluated sample obtained has a lower probability than alpha”, but from the other direction.

One perspective on this comes from a CI: if my CI is narrow (due to high power) and includes zero, I know that only for a small range of hypotheses centred around and including zero, the probability of obtaining the current (or a smaller) measure would be less surprising than my alpha; conversely, for all hypotheses assuming an effect outside of my CI, the data would be surprising. But I am not sure how to phrase this in reference to power.

Admittedly, this question could probably be answered by reading Cohen 1988, however, I do not have the book with me. Also, I assume this problem is commonly enough misunderstood. I would be happy with a pointer to an authoritative source, too.

Improve this question
$\endgroup$

4 Answers 4

Reset to default
2
$\begingroup$

I'm not sure why you're bent on using power and alpha in your statement when you already have a CI that's narrow and around 0. From that you can argue that this range of small values close to zero is where you believe the effect likely is and that effects of greater magnitude are unlikely. It's implicitly using your power and alpha, you're just saying it in a much more concise and easily interpretable way.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Mostly because I am trying to understand precisely what it means when Cohen writes: "the conclusion is justified that no nontrivial effect exists, at the P = .05 level". I do think a CI of the effect size is probably the superior approach (though on the other hand, power is easily comparable between studies, but the width of a CI is not, isn't it?). $\endgroup$
    – jona
    Commented Jul 27, 2013 at 14:07
  • $\begingroup$ What you will find in every answer to this is that you run into multiple theoretical, and lay, interpretations of all of your words that make a simple statement nigh impossible. Don't bother attempting it. Power is comparable between studies all by itself, but it's meaningfulness is questionable without effect sizes which now is starting to get to a CI. CI's are nice and comparable, that's narrower, wider, captures 0, doesn't capture x, etc. CI's are comparable between studies. There are even inferential rules about it and they're used a lot in metanalysis. $\endgroup$
    – John
    Commented Jul 27, 2013 at 17:25
  • 1
    $\begingroup$ Although Bayesian and likelihood approaches are superior in my view, for now I would just ignore the $P$-value and use the 0.95 confidence interval. The confidence interval does not need a null hypothesis to be interpretable. $\endgroup$
    – Frank Harrell
    Commented Jul 28, 2013 at 12:05
3
$\begingroup$

Part of the difficulty might result from the conflation of the Fisher and Neyman-Pearson approaches to testing.

As far as I understand the problem (and I haven't read the original sources and certainly don't claim to be an expert), interpretation of the p-value and the “data more extreme” and “or something unlikely happened” language comes from Fisher but he has no concept of power and not much to say about experiments that fail to reject the null hypothesis.

On the other hand, in the Neyman-Pearson framework, interpretation is strictly about long-run frequencies and there is really nothing to be said about a single experiment. Under that view, the only thing you can say is that if the effect size were actually d, the test would lead you to accept the null hypothesis at most 5% of the time (that's $\beta$ or 1 - power). Both $\alpha$ and $\beta$ must be set beforehand.

Your description of a significant result feels quite Fisherian, even if it mentions $\alpha$. This would suggest that it is impossible to create a rigorous analogous statement about non-significant results as it requires the concepts of power and type II error ($\beta$), which only completely make sense in the Neyman-Pearson framework.

It's precisely because all this is quite frustrating that it's very difficult to avoid any pseudo-Bayesian interpretation (optionally prefaced with a “that's not quite right” if you want to show some sophistication but still don't know what else to say).

But like I said, I am not sure that I fully grasp all the issues here and I am happy to be corrected!

Some literature on this (not sure how “authoritative” you would consider it):

  • Hubbard, R. (2004). Alphabet soup: Blurring the distinctions between p's and $\alpha$'s in psychological research. Theory and Psychology, 14 (3), 295-327.
  • Hubbard, R., & Lindsay, R.M. (2008). Why P values are not a useful measure of evidence in statistical significance testing. Theory and Psychology, 18 (1), 69-88.
Improve this answer
$\endgroup$
2
$\begingroup$

One possible way of interpreting any statement about a p-value, significant or not, relies on the definition of a p-value. Suppose, for illustration, that p is 0.6. Then:

If, in the population from which this sample was drawn, the effect size was 0, then we would get test statistics at least as extreme as the one we got in this sample 0.6 of the time.

This does not depend on power; power comes into it in that, if there is high power, then only small effects will be non-significant.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I am not satisfied with this solution since it does not apply the additional information contained in the power calculation. Conceptually, power informs us how much our result depends on n; simply looking at the p value, it may just be far from 0 because a small sample is not reliably representative. Phrased differently, high p in a low power study only says that even if the nil-null is true, values different from 0 are not unexpected in small samples such as the given one. Right? $\endgroup$
    – jona
    Commented Jul 27, 2013 at 13:35
  • $\begingroup$ That is part of my answer already: The combination of effect size and significance incorporates power. If a large effect size is not sig., there must be low power; if there is high power and an effect is not significant, it must be small. $\endgroup$
    – Peter Flom
    Commented Jul 27, 2013 at 14:06
1
$\begingroup$

In a test with b power to detect an effect of size d at alpha = a, if the test does not reject the hypothesis, we may state that the real population effect is possibly less than d. However, power is not an estimate of the reliability of this estimate (high power does not allow one to prove the null).

Rationale: power is defined so that if a test has b power to detect an effect d at alpha = a, b% of experiments that are investigating a phenomenon where the effect in the population is d will result in samples that lead to a successful test. Conversely, (100-b)% of samples drawn from this population will yield tests that do not reject the hypothesis. We are not justified in inferring from the power anything regarding a possible population for which the real effect is smaller than d, because power only relates to what happens when we test samples from a population where the effect is just d. We do still feel somewhat more confident in inferring from a failed test that the effect is below a d for which we have high power than for a d for which we have low power. However, this is only for the reason that we are reducing the chance of failed tests for real effects (type II errors), not because our power for detecting an effect of size d informs us in any way about populations where the effect is not d. In bullet points, there are 2 possible cases for when a test fails to reject:

  • a population where the effect is d, but we are observing one of the (100-b)% cases where the test fails
  • a population where the effect is not d

High power reduces the likelihood of the first set, but is unrelated to the likelihood of the second event.

(This answer comes from a thorough explanation in chat by @john which ultimately led me to accept that yes, CIs ARE the answer.)

Improve this answer
$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.