0
$\begingroup$

I'm trying to analyse bullying experiences across three age groups. The DV is scored on a 5-point Likert, and the IV is categorical (ages 11, 13, and 15).

Initially I ran an ANOVA to see if there was a significant difference in bullying experiences across three age groups. The results came back as non-significant with a very small effect size. I re-ran it using a Kruskal-Wallis because it was ordinal data, and again found no significance and a very small effect size. Finally, I tried three Mann-Whitney U tests with Bonferroni corrections, and found the same. Normally I'd call it quits there and accept the results, but when I look at my table of means, there's quite substantial differences between the three groups. I'll summarise one example below:

Age cat. Mean SD N
11-yrs 1.88 1.18 49
13-yrs 2.38 1.65 64
15-yrs 2.62 1.62 58

Would it not be logical to assume there's some difference between 1.88 and 2.62 when it's only on a 5-point?

My main question here is if the large standard deviations can be responsible for this inconsistency?

In this example 32/171 participants gave the highest score of 5, so it didn't seem like an outlier.

Improve this question
$\endgroup$
1
  • $\begingroup$ Welcome to CV, Hannah. These differences in means are sufficiently great relative to the SDs that I don't believe your original results. When I run this ANOVA I find p = 0.042, which is inconsistent with a claim of "no significance." Could you show the ANOVA output? $\endgroup$
    – whuber ♦
    Commented Apr 4, 2023 at 12:43

1 Answer 1

Reset to default
1
$\begingroup$

The first thing I am seeing is that your variances and sample sizes are not equal between groups. One or the other must be equal (or close to it) and the data must be normally distributed. I would suggest running Bartlett's Test to confirm the variance is sufficiently similar. But your sample sizes are pretty different in size. The largest is 30.6% larger than the smallest. How to Perform an ANOVA with Unequal Sample Sizes 2 sample T-Test for Unequal Variances Bartlett's Test

But to answer your question, significance is sort of impacted by SD as the standard error is the SD/sqrt(n), but given that SD is an intrinsic property of the sample population, it's not something that you can change. To increase the likelihood of significance, you need to increase the sample size. However it's also entirely possible that its just not significantly different. As you didn't provide the test statistic, it's hard to say if there's an error. Regardless, I would flesh out the possibility of any issues relating to variance and sample sizes being unequal first. If they aren't equal, use the non-parametric equivalent test (kruskal-wallis-test (non-parametric version of ANOVA).

One last thing, the amount of people who selected 5 may not be an outlier, but at nearly 20% of the total sample, I suspect the distribution doesn't resemble a normal distribution (not that you can have normal distributions with Likert scales, technically). If the distribution is skewed, try Levene's Test.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Although these are good points, the first paragraph is a little misleading IMHO, because (a) the group sizes are reasonably close to each other ("30.6% larger" is not a relevant way to compare group sizes for an ANOVA) and (b) given these substantial group sizes and the bounded response values, the sampling distributions of the means really will be close to Normal. Thus, these factors do not explain the claims in the question. $\endgroup$
    – whuber ♦
    Commented Apr 4, 2023 at 12:46
  • $\begingroup$ Ok then, what would be a relevant way to compare group sizes for anova? As for (b), apparently, I need a nap because, yeah, for some reason I was thinking this was population data while somehow also ignoring the fact that data for an entire population doesn't require inferential statistics. Oops. Thanks for catching that. $\endgroup$
    – JewJitsu11B
    Commented Apr 4, 2023 at 14:25
  • $\begingroup$ A relevant way to compare group sizes would be to consider how they are related to uncertainties in variance estimates. That would indicate generally that, for instance, an imbalance of 100 in one group and 1000 in another would not be anywhere near as severe as an imbalance of 2 vs. 20. As far as comparing group variances goes, use chi-squared distributions as a reference. They generally indicate that when the ratio of the largest to smallest variance is under 3:1 or so you are OK (there's little evidence of heteroscedasticity), depending a little on the group sizes and number of groups. $\endgroup$
    – whuber ♦
    Commented Apr 4, 2023 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.