# Teaching students about non-significant results and large effect size

This year I am going to teach statistics to sophomore year students of psychology. We'll be training such methods as one-way ANOVA. The example will be the time-reaction of a cognitive task among alcoholics, non-alcoholics and ex-alcoholics. This is just an made-up, imagery-stimulating example.

I'd like to ask you all, what I should tell them when it comes to the interpretation of the statistically non-significant results with a large effect size.

$$F(2,21) = 3.14; p = 0.22, \eta^2=0.30.$$

What would you say to a psychology student about how to understand these findings?

A) The results are statistically non-significant F(2,21) = 3.14, p-value = 0.12, eta = 0.20 thus the researcher's hypothesis (about the influence of alcoholism on working memory) is rejected.

B) Although the results are statistically non-significant F(2,21) = 3.14, p-value = 0.20, due to a large effect size $$\eta^2$$ = 0.20 the research hypothesis may be correct (thus further examination needed).

C) The results show that the data are consistent with null hypothesis of no-effect (of an alcoholism on a capacity), F(2,21) = 3.14, p-value = 0.20, but the large effect size $$\eta^2$$ suggests that the research hypothesis may be correct.

D) … anything else?

• If you replace "is disconfirmed" by "cannot be confirmed", A) is fine. B) and C) actually suggest that the chosen study design was wrong (too small sample). You can avoid problems with large effects and insignificance by clearly distinguishing between descriptive results (here eta-squared) and inferential stats. – Michael M Sep 7 at 16:29
• Frank Harrell and others recently started a discussion about the best way of communication of (frequentist) study results. You might find some inspiration in the section "Interpretations of So-called 'Negative' Trial". – COOLSerdash Sep 7 at 16:53
• @COOLSerdash I think I read that users shouldn't write 'thanks' but maybe "thank you very much" is much proper :-) Thank you very much. I didn't know that. – Lil'Lobster Sep 7 at 17:00
• Is it possible to report a confidence interval for the effect size, not just a point estimate? I would start there just to convey that the data are compatible with a range of possibilities. Your students should develop the notion that clear cut answers are not always possible and - sometimes the answers can be inconclusive or even contradictory. See daniellakens.blogspot.com/2014/06/… for ideas on how to get a confidence interval for eta squared. – Isabella Ghement Sep 7 at 17:12
• Now I have to challenge you back to really create an example where you did perform the sample size calculation upfront and came up with the said confidence interval for eta squared! 😜 But it is not inconceivable that something may have gone wrong with your assumptions in the initial sample size calculation (e.g., size of effect assumed was wrong) and you ended up getting insufficient power for the appropriate effect size. Statistics is an art after all, just as much as it is a science. – Isabella Ghement Sep 7 at 17:19

I wouldn't explain it to them in any of these ways. (Note also that the numbers in some of those explanations are wrong -- you need to be more careful of that)

A) The results are statistically non-significant F(2,21) = 3.14, p-value = 0.12, eta = 0.20 thus the researcher's hypothesis (about the influence of alcoholism on working memory) is rejected.

You can't support this statement -- the absence of evidence of the presence of the predicted effect is not the same thing as evidence the effect is absent.

C) The results show that the data are consistent with null hypothesis of no-effect (of an alcoholism on a capacity), F(2,21) = 3.14, p-value = 0.20, but the large effect size η2 suggests that the research hypothesis may be correct.

If you're trying to show them how to construct a conclusion I'd say the broad form of C is closest to being reasonable (though I think for that purpose you could do better as well), but for an explanation I'd lean more toward something roughly along the lines of

"The results are consistent with the absence of an effect, but the estimated effect size is large; this occurs because the standard error of the effect size is also large -- we can't tell if the population effect is big or small from these data; the sample size is too small to estimate it well."

I'd also add that reaction times are not going to be anywhere close to normally distributed in general; typically they're quite skew and tend to have spread related to the mean (smaller means are associated with smaller standard deviations). I wouldn't be using regression/ANOVA with reaction times; I'd be choosing a more suitable model.

Excellent start with @Isabella's Comments. Here are some fake data that might be sufficiently similar to yours for purposes of discussion. I don't know if you use R statistical software, but other kinds of software will do about the same thing for data like these.

set.seed(907)
a = rnorm(8, 13, 7)   # alcoholics
n = rnorm(8, 15, 5)   # non
x = rnorm(8, 20, 5)   # ex
y=c(a,n,x)                 # data in stacked format
g=as.factor(rep(1:3, 8))   # group number 1, 2, 3


First look at descriptive statistics, Maybe review what the DF's mean and what information goes into SS(Group) (group means) and into SS(Error) [group variances]. Then what divisions lead to the F-statistic. Then pick the ANOVA table apart.

summary(a); sd(a)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
3.655   8.117  12.896  12.464  15.293  20.847
 5.919241   # sd of gp a
summary(n); sd(n)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
3.597  11.780  15.221  14.672  20.023  21.437
 6.276581
summary(x); sd(x)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
11.60   14.68   17.28   17.15   18.01   25.16
 4.046853


Respective means are about 13, 15, 17, so there might be an effect. Medians show a similar pattern.

Look at stripchart (or dotplot) of data. Do you see any difference? Why don't the differences among means show more clearly?

stripchart(y ~ g, pch=19) Maybe also look at boxplots. Large variability obscures differences in medians.

boxplot(y ~ g, col="skyblue2", horizontal=T) Now look at ANOVA table.

anova(lm(y ~ g))

Analysis of Variance Table

Response: y
Df Sum Sq Mean Sq F value Pr(>F)
g          2  16.65   8.324  0.2472 0.7832
Residuals 21 707.07  33.670

• Residual (Error) line has huge Sum of Squares, so large Mean Square. F-statistic is ratio of MS(Group) to MS(Resid). So large MS(Error) makes the F-statistic small, hence P-value large, leading to a non-significant result.

• If you had more data, then DF's would be larger, MS(Error) smaller, F-larger, P-value smaller.

• Alternatively, if group variances were smaller, then MS(Error) might be small enough for a significant effect---even with the current sample sizes.

Here is a plot of the distribution F(2, 21). The observed value of $$F$$ is shown as a vertical dotted line. The P-value is the area under the density curve to the right of the dotted line. $$F(2,21)=3.14;p=0.22,η^2=0.30.$$

D) anything else

The null is "accepted," that there probably is no effect. What does "non-rejection" mean?

It can mean two things. First, it could mean you have no knowledge regarding the effect. If you took R.A. Fisher's understanding of a p-value, then the absence of rejection is the same as no added knowledge.

If you approached this from a Pearson and Neyman decision-theoretic perspective, then it means that you should behave as if there is no relationship. From a Pearson and Neyman inferential perspective, however, you simply have no findings.

The p-value gets knocked around quite a bit because people abuse it and ignore effect size.

What it is telling you is that despite the possibly random appearance of a large effect size, it still cannot be shown that it is different from zero.

Imagine I am a rookie major league baseball player and I am batting 800 with five at-bats. My effect size is HUGE, but I cannot falsify that my real batting average is 400 $$(p<.058).$$

The non-rejection of the null with a large observed effect could mean many things.

First, it could mean there simply is no effect. If there is literature regarding this that there is an effect, then further investigation is warranted. The result could be a false negative. If there is no literature, it could mean that everyone is getting non-significant findings and so no editor is publishing the findings. So it could be that there is no effect.

Second, there could be poor hypothesis or experimental design. It could be that there is an effect but either the hypothesis or the experiment was poorly constructed. For example, in a famous case of Yule's paradox, UC Berkley found that its admissions were gender-biased against women. In fact, the effect was strongly supported though the effect size wasn't necessarily that large. A deeper investigation determined that admissions were gender-biased against men, though again the effect size was small.

The poor experimental design resulted in a conclusion that was the opposite of the facts.

It could be that the experimental design was too simple. For example, there may be some hidden bias among who is an ex-alcoholic OR there could be a hidden bias among who said "yes" to participate. It is possible that gender matters. It is possible that a whole range of things actually matters, but that the division points into the three groups are a poor division.

It could mean that ANOVA was the wrong test. For reaction times, there is a reasonable argument to be made that assumptions are violated.

The difficulty is that Frequentist methods of statistical analysis assert that the null is 100% perfectly true. If you reject the null, then to some degree of confidence, you can assert it is not a true statement. However, if the null is not rejected, that does not imply that it is true.

Frequentist methods of statistical analysis are a probabilistic form of modus tollens. If A then B, and NOT B, therefore NOT A. However, sentential logic does not imply "If A then B, and B, therefore A." In the case of B is true, then A can be true OR false. While it is true that if A is true then B must be true, we know B is true. If A is false, B can still be true. That is why Fisher held that failure to reject the null was the same as having no knowledge about the topic.

Logically, non-rejection of the null implies that either A is true OR A is false. The effect size is irrelevant to that discussion. Since it is a tautology that A is true or false, what you do should not depend on the results of your experiment.

If you believe that there should be an effect, then you could, for your own personal utility, expand your investigation. If you do not believe there should be an effect, you could, for your own personal utility, not continue to investigate. However, the results should not factor into your decision. It is as though you had never decided to look in the first place.