Comparing means in large samples: effect size and p-value

Question

I have a large sample (10000+ per location) of temperature measurements which I would like to compare between 3 locations. I'm checking if there is a significant difference in temperture means between these 3 locations. My data fails the normal distribution and homogeneity of variance assumptions, so I did a Kruskal-Wallis test (it is significant in all pairwise comparisons; p = 0,000) and I calculated the effect size, which was 0,081. Due to a large sample of data, the p-value was expected. My question now is how to interpret or better yet, present the results? I presume that the effect size is considered low in my case, so there is a difference but it is not that "significant", even though the p-value is very low? Can I say that?

I'm comparing these 3 locations (which are generaly very similar), because a certain plant species grows only on one of them and I'm checking if any of the abiotic factors could be contribuiting to the fact that the species grows only there. Based on my results I cannot really say that temperature is one of those factors?

I got the effect size value as in this video: https://www.youtube.com/watch?v=Rna4VHHu4cI - is this the correct method? I'm on a basic level in SPSS.

Thank you in advance!

Answer 1

Normality testing with this kind of sample size (or in fact any sample size) is not a good way to pick your analysis method. It is pretty likely to reject a normal distribution (which you hopefully - if at all - checked for the residuals of a regression model that correctly takes correlations into account, not the raw data), even if the deviation are irrelevant for using a normal distribution based method.

If the data are for the same time and date (if that is not the case, then you need to check that you are not getting differences simply due to measuring at different times), then this is actually paired data for those datetimes, plus there is likely correlation over time. If you use methods that do not reflect this, you will understate uncertainty and have p-values that are inappropriately too small. You may think you have 10,000+ observations per location, but depending on the correlations, you probably effectively don't. If you read the temperature 10,000 times within 1 second, you would effectively have barely more than 1 observation per location. Presumably your case is somewhere inbetween, but the p-values you've got since far are completely invalid.

You probably need a model that reflects paired measurement occasions in time (e.g. random effect on the intercept for that) with either those or the residuals correlated in some way over time (something as simple as AR(1) is usually inappropriate) and models the time series of temperatures (or simpler, you could take the predicted temperatures for the region from some wheather service that gives sufficient granularity in time - e.g. by hour with you smoothing that out to minutes). You can then put a location effect into the model and test for that. You may also wish to consider whether there might be interactions - e.g. time of day with location (e.g. one location may be shaded by trees for longer on the morning than another).

Finally, "statistically significant" does not equate to an effect having a meaningful size, so one may certainly say that while some difference was statistically significant the effect size was not of practical relevance.

Answer 2

My question now is how to interpret or better yet, present the results? I presume that the effect size is considered low in my case, so there is a difference but it is not that "significant", even though the p-value is very low? Can I say that?

You can say that "In analysis of temperature records in three locations, the effect of the location was statistically significant (p = xxx). Eventhough the effect size was small (xxx), its direction was same as predicted in our hypothesis." Or something along these lines. Key issue is the direction of the effect, and whether it supports your hypothesis and story or not.

Based on my results I cannot really say that temperature is one of those factors? Based on the results of the analysis, it was statistically significant, so you could say that it was one of the factors (if direction of effect correct). Though you probably should check about the homogeneity of variances: some sources say that its violation could undermine the results/interpretation of Kruskal-Wallis test.

Answer 3

There are no generally accepted interpretations for epsilon-squared for the kruskal-wallis test. At this link, I have some guidelines for interpretation but they are in no way authoritative. 0.08 comes out in the "medium" category. My guess is that no one on the planet has an intuition about how to interpret a kw epsilon-squared value. It's fine to report as a matter of course. But I would recommend reporting something more meaningful for your readers. The Vargha and Delaney's A for the two most different groups is easy to explain and interpret.