2
$\begingroup$

I am running a one-way ANOVA to compare a numeric continuous variable between four groups, and getting a very high F-value. Below are the descriptives and code I used in R (I get similar values when trying a different package in Python):

> oneway.test(ideology_score ~ region, data = mydata)
   One-way analysis of means (not assuming equal variances)

data:  ideology_score and region
F = 1270.6, num df = 3, denom df = 101578, p-value < 2.2e-16

Total N of observations = 262,267

Group Mean SD N
Group1 -0.43257859 1.285637 80,136
Group2 -0.13539030 1.355517 26,915
Group3 -0.08337834 1.362262 85,512
Group4 -0.39797591 1.263706 69,704

The means and standard deviations of the variable are almost the same for each group, and the variance and distribution of scores within each group are quite similar as well.

Is there anything else that may be going wrong here to explain the high F value?

$\endgroup$
3
  • 3
    $\begingroup$ At the very least, report the number of observations you have! $\endgroup$
    – whuber
    May 16 at 21:01
  • $\begingroup$ Can you show the means and standard deviations as well? And tell us how you've obtained this F? Right now it's just English words, hard to figure out anything... $\endgroup$ May 16 at 21:19
  • $\begingroup$ thanks for the suggestion, updated with more information. $\endgroup$ May 16 at 21:30

1 Answer 1

5
$\begingroup$

With sample sizes as large as those you will almost always find that the statistical test gives very low p-values. Divide those SD values by the square root of the sample sizes to see the estimated SD of the means (i.e. the SEM) and you will see that they are tiny compared to the various differences between the means. That gives you the high F-value.

You need to decide on a way to deal with the fact that statistical significance does not correspond to real-world significance. What you need to decide is whether a difference between, for example, -0.43 and -0.13 is of any importance. It might be trivial, or not.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.