1
$\begingroup$

I have a sample size of 23, with 11 and 12 participants in each group. I conducted t-tests for several continuous dependent variables. All of them are non-significant, but some of them have quite high Cohen's d values (for example 0.6 or above). In addition, the confidence intervals in these cases, while still spanning zero, usually have one case which is very very close to zero.

As an example, one of the tests has a Mann-Whitney U value of 41, a p-value of .11, CIs of -0.00004 and 1.2, and cohen's d of 0.75. How should I interpret this finding? Intuitively I would say it is a power problem, but with medium-large effect sizes I am not sure, since even small samples would theoretically be enough to detect such effects.

I used the jamovi package in R to conduct the tests

ttestIS(data = dat, vars = vars(variable_1, variable_2, variable_3, variable_4, variable_5), group = condition, students = FALSE, mann = TRUE, meanDiff = TRUE, desc = TRUE, effectSize = TRUE, ci = TRUE, plots = TRUE)

Update: I went off the effect size interpretations specified by Cohen (0.2 = small, 0.5 = medium, 0.8 = large). The dependent variables are self-report questionnaire items on a 5 point likert scale. The responses were non-normally distributed, hence the use of Mann-Whitney U tests.

$\endgroup$
4
  • $\begingroup$ Cohen's d "as large as" > 0.6 is highly specific of the domain, IMHO. Since you mention statistical power you're certainly aware of the fact that it's bad to confuse the magnitude of an effect and its statistical significance. We would probably need more information on your experimental setup and the distribution of your response variable to provide useful answers in this case. $\endgroup$
    – chl
    Nov 3 '20 at 12:02
  • $\begingroup$ @chl the domain is in psychology, so I was going off the effect size interpretations initially specified by Cohen :) The dependent variables are self-report questionnaire items on a 5 point likert scale. The responses were non-normally distributed, hence the use of Mann-Whitney U tests. I've also updated my question with this info accordingly. $\endgroup$
    – becbot
    Nov 3 '20 at 12:19
  • $\begingroup$ There is a difference between statements about the specific sample and statements about the population. $\endgroup$
    – Michael M
    Nov 3 '20 at 12:36
  • $\begingroup$ Note that if you consider your question answered, you should accept one of the answers (checkmark below the score), otherwise you should specify what is missing in the present answers in order to satisfy your question. $\endgroup$
    – Kuku
    Nov 3 '20 at 15:07
4
$\begingroup$

Short answer: You don't.

Since your effect is not significant (you fail to reject the null hypothesis that there is no effect), if you're following the rules of null hypothesis significance testing, you cannot conclude that there is any effect here.

Additional points

  • You're running 5 tests, without adjusting for multiple comparisons, so the chances of making a type 1 error here are quite high. This makes it even more likely that your large effect size is large just by chance.
  • The wide confidence intervals show that even though the effect size might be large (and might even be larger than 0.75), it might also be small, zero, or negative, since your data does not rule out these possibilities.
$\endgroup$
2
  • $\begingroup$ I am confused by the wording "This makes it even more likely that your large effect size is large just by chance." How does the number of tests affect the effect size itself? I agree with the main point however. $\endgroup$
    – einar
    Nov 3 '20 at 12:47
  • 1
    $\begingroup$ I'm being a bit sloppy, since this is a beginners question, and the correct NHST definitions are obtuse: "The chances of obtaining, due to sampling error, a large effect size in cases where the null is actually true increases as you run more tests". $\endgroup$
    – Eoin
    Nov 3 '20 at 13:00
2
$\begingroup$

It sounds like you are replying to your own question already. The effect size statistic is independent of your statistical significance. Here a extreme example you can run in R:

data <- matrix(c(1,0,2,0,1.5,0,
                 1.7,1,1.8,1,2.2,1), nrow= 6, ncol = 2, byrow = TRUE)
data <- as.data.frame(data)
data[,2] <- as.factor(data[,2])


effsize::cohen.d(formula = data[,1] ~ data[,2], data = data)
t.test(data[,1] ~ data[,2])

We have only three observations per group, so before even estimating anything we know it is underpowered and a practically useless analysis. Despite this, the output gives a Cohen's d of 4.05, an extremely large effect. This says nothing about our sample size, our power, or its significance, hence there is no conflict in your results. The two-sample t-test gives a p-value of 0.3071, unsurprising given that we just have three samples per group.

You can see Cohen's D statistic just as a standardized difference between means of distributions, and that says little about how stable those distributions are or the quality of your sampling. The image below shows the point: the distributions have great variability and a large overlap, but the distance between their means (the vertical lines) is relatively large, hence the high p-value and high effect size.

Two broad densities, with vertical lines showing the  mean of each

Also worth noting what Eoin says on the other answer: you are doing multiple comparisons with no correction for it. This can be seen as a way of p-hacking or torturing the data if all you care about is finding a "significant" result, as it seems to be suggested.

$\endgroup$
2
  • $\begingroup$ Thank you for the example! You and @Eoin both raised good points about running tests without corrections - I know for t-tests Bonferroni can be used but I didn't look into Mann-Whitney U yet. I was mostly confused about why the results would still demonstrate a 'large' effect size, since I know small effect sizes can still be significant but I didn't know about the other way around. $\endgroup$
    – becbot
    Nov 3 '20 at 13:10
  • $\begingroup$ You can see multiple comparison correction as an issue in hypothesis testing in general, not specific to a method like a t-test. How much of a serious fault is this will depend on the specific context of what you are doing though: is this a class exercise? is this actual research? It's common for students to be implicitly taught to torture the data until they find a "reportable" p-value: this is bad science. $\endgroup$
    – Kuku
    Nov 3 '20 at 13:16
1
$\begingroup$

All of them are non-significant, but some of them have quite high Cohen's d values (for example 0.6 or above)

The fact that the effect size is large doesn't necessarily mean that a test for no-difference will return a tiny p-value. Here's an example:

x1
1.46  1.62  0.77  0.38  0.92  0.36 -0.68 -1.06  0.22  0.89
x2
-1.11  0.38  1.18 -2.25  0.53  0.61 -0.47 -0.45 -0.46 -0.79

If I'm not mistaken, Cohen's d for the difference between mean of x1 and x2 is 0.83 but a t-test returns p=0.08 (not very convincingly different from 0):

t.test(x1, x2)

    Welch Two Sample t-test

data:  x1 and x2
t = 2, df = 20, p-value = 0.08
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.099  1.643
sample estimates:
mean of x mean of y 
     0.49     -0.28 
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.