Suppose you have 3 groups, 'Atheists' ($n_1=102$), 'Nones' ($n_2=178$) and 'Religionists' ($n_3 = 390$). You want to find out if there is a difference in the experience of meaningfulness between these groups. Further you want to know if there is different subtypes of atheists, and the differences between these, if any. So you give your participants a Questionaire, which is supposed to measure different sources of meaning (26 in total) as well as degrees of experienced meaningfulness and crisis of meaning.

  1. A MANCOVA reveals now there is a "highly significant overall effect of group differences", with $F(4, 1272) = 4.78, p = .001, η^2 = .015$, according to which Atheists experience less meaningfulness than Religionists and Nones
  2. k-means cluster analysis reveals 3 clusters of Atheists $F(52, 148) = 5.74, p < .0001, η^2 = .67$
  3. These clusters show significant differences in both meaningfulness, $F(4, 196) = 9.25, p < .0001, η^2= .16$ and frequency of crisis, $\eta^2=.2$
  4. You compare sources of meaning (26 items) of each cluster to Religionsts and Nones. MANCOVA, controlling for sex, age, education, and family status results in overall effects of $p < .001$ and $η^2$ of $.21, .29$ and $.29$ for each cluster. Regarding the items, you get $\eta^2$ varying from $0.01$ to max. $0.2$, depending on the source of meaning and cluster.

My actual question comes now: What should I think about these results? $\eta^2$ of .67 regarding clusters seems quite good. But the rest? I mean throughout the whole paper it's always stated that the resulsts are significant, but it's never mentioned that there is effect sizes as low as $\eta^2= .01$. One result for a subgroup of atheists, for example, is that 30% experience crisis of meaning compared to 4% within Religionists, even though $\eta^2=.03$ only in this case. It's just completely ignored. Is it right to say in this case that such crises are more frequent within this cluster? I mean 30% is really a lot and p<.001 is really low, but so is also $\eta^2 =.03$.

Since it's an academic paper, I'm not sure if it's really just not important and I conceptually misunderstood something regarding the effect size.

I've done some reading the last few days though and found that most people (scientists?) fail to correctly interpret their results, focusing too much on NHST while ignoring effect sizes.

Now from what I've read so far, I would not state, that there is a practically significant difference between these groups, even though it might be statistically significant. Also I've read with "high enough" sample size, it's not that hard to find staistically significant differences. Should this already be considered in this case? (And when is $n$ "high enough" btw?)

I have to admit, I'm also still confused regarding interpreting effect sizes. I'm already on a more technical paper I've chanced upon on this forum but I haven't had the time to finish it yet. I also checked out this page, linked in another post here.

I'm supposed to give a presentation about that paper cited above tomorrow. My thoughts here are basically points of criticism I've wanted to mention (especially since my lecturers also never mention effect sizes!), so I would really appreciate any help and clarification on this topic.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.