8
$\begingroup$

I am writing my Master's thesis on adult's performance on cognitive and linguistic measures. I have four age groups and am investigating if there is an age difference in the performance on summarized measures and individual tasks.

I ran a Kruskal-Wallis one-way ANOVA and found one statistically significant result out of 15 tasks, on a task called "Design Generation" with $p = 0.003$. Dwass-Steel-Critchlow-Flinger pairwise comparisons showed me between which two pairs of age groups there was a difference. All good!

However, DSCF also showed me a significant difference between two age groups on a measure (Design memory) where Kruskal-Wallis did not identify any significant differences initially.

My question is - is DSCF a post-hoc test which is supposed to only be applied when statistically significant differences are found on Kruskal-Wallis? How should I report these findings? Is it wrong to report that I found something significant using DSCF on a measure where I did not find anything significant using Kruskal-Wallis?

$\endgroup$

3 Answers 3

12
$\begingroup$

My question is - is DSCF a post-hoc test which is supposed to only be applied when statistically significant differences are found on Kruskal-Wallis?

Yes, the Kruskal-Wallis test is a non-parametric omnibus test which compares three or more independent samples. It assesses whether the population medians on a particular measure are the same across all groups. It is the non-parametric equivalent of the one-way ANOVA, and as with ANOVA, importantly, the K-W test does not indicate where the differences lie between groups, just that at least one group is different.

The Dwass-Steel-Critchlow-Fligner (DSCF) test is a non-parametric procedure that can be used post-hoc after a K-W test. It is designed to identify which specific pairs of groups have significant differences. Note that this test adjusts for multiple comparisons to control the overall Type I error rate, which is a good thing here.

If the K-W test shows no overall significant difference but DSCF reveals differences between specific pairs, it suggests that the K-W test may not have been sensitive enough to detect subtle differences between specific groups. DSCF is more focused and can detect differences that the broader Kruskal-Wallis test might miss, while adjusting for multiple comparisons.

This issue has been the subject of much debate on this site, and you can easily find conflicting advice. For example, in the accepted answer here, LSC writes:

In general, if you use an omnibus test, such as an ANOVA F-test or a Kruskal-Wallis H-test, it is illogical and poor practice to conduct pairwise comparisons when you fail to reject the null hypothesis on the omnibus test. Conducting the comparisons flies in the face of the omnibus: insufficient evidence to conclude differences does not warrant further investigation, as a general rule.

Usually, I would say report analyses you run, but in this case (which is different from selective reporting), the post-hoc p-values are inappropriate to interpret and should be omitted. The omnibus p-value is appropriate since this is the “gatekeeper” test.

, while in the heavily upvoted answer here, Harvey Motulsky writes:

Since multiple comparison tests are often called 'post tests', you'd think they logically follow the one-way ANOVA. In fact, this isn't so.

"An unfortunate common practice is to pursue multiple comparisons only when the hull hypothesis of homogeneity is rejected." (Hsu, page 177)

Will the results of post tests be valid if the overall P value for the ANOVA is greater than 0.05?

Surprisingly, the answer is yes.

and he goes on to explain that the omnibus test is a test of the the null hypothesis that all the treatment groups have identical mean values, so any difference you happened to observe is due to random sampling. Each post-hoc test tests the null hypothesis that two particular groups have identical means. The post tests are more focused, so have power to find differences between groups even when the overall ANOVA reports that the differences among the means are not statistically significant. [here he is talking about post-hoc tests following ANOVA, but the argument applies well to K-W too]

My own advice is to consider what your research question is, along with the analysis you want to conduct before you collect the data. If your research hypothesis requires pairwise comparisons, and not an overall/omnibus test, then don't run the omnibus test first - the pairwise tests can be the primary analysis. In your case, it is too late for that, so I would suggest, when reporting your findings that you explain both the results of the Kruskal-Wallis test and the DSCF tests. Acknowledge the overall test result and then discuss the specific pairwise differences found in the DSCF tests.

Reference: Hsu, J. (1996). Multiple comparisons: theory and methods. CRC Press.

$\endgroup$
3
  • $\begingroup$ Thank you for your answer. It makes sense that some small differences will only be obvious when pairwise comparing for example the youngest and oldest group, not when looking at all groups. In my case I should have only ran DSCF as you said, as that is the information I need to say anything valuable about age and performance. As for why I ran DSCF after K-W: I am using jamovi (the only program used in our course). Jamovi only allows DSCF after K-W and runs DSCF on all measures, whether or not K-W identified any significance. $\endgroup$ Nov 19, 2023 at 13:51
  • $\begingroup$ @Sam ahh ok I understand. In that case I would suggest explaining in your thesis that you ran the ominbus test first because that was the only option you had in order to run the pairwise tests due to a limitation in the software. $\endgroup$ Nov 19, 2023 at 14:42
  • 3
    $\begingroup$ K-W doesn't exactly test medians but the answer is otherwise brilliant. K-W contains a perfect multiplicity adjustment. But there is a much more serious problem here. The OP had to look at 15 variables to find one that was "significant". No multiplicity correction was made for the 15. If you only find 1/15 and don't want to do a proper multivariate analysis, I would not recommend analyze the one. $\endgroup$ Nov 19, 2023 at 14:57
9
$\begingroup$

What is ANOVA?

It is important to note some of the reasons that ANOVA exist, as it can provide some context about how you should consider your omnibus ANOVA and the pairwise tests that accompany them (and largely your question about interpreting both).

The person largely credited for its invention Ronald A. Fisher was hired at an agricultural facility to determine differences in crop yield variation in plots at a farm. The facility had collected nearly a century's worth of data on this (about 70 years by the time Fisher got there) that included fluctuations in weather patterns, which fertilizers were used, soil deterioration, etc. This was a lot of info that seemed to only convey that year-by-year, some fertilizers worked and then some years they didn't. There was a lot of random noise that people simply didn't consider at the time as confounding effects (Fisher, 1921).

Fisher then came up with the solution of finding a way to systematically apportion effects attributed to the actual treatment (the fertilizers) and effects that were simply random (weather, etc.) by tabulating the data and then determining the mean squares between and within groups (see original table below for how simply inspecting the differences here wasn't super informative):

Table I

Formulation of Parametric and Nonparametric Versions

The omnibus test of the ANOVA is simply stated as:

$$ H_0: \mu_1 = \mu_2 = \mu_3 $$

where the $\mu$ here is simply a sample mean and can be theoretically extended to any number of groups. Note that for the nonparametric version you use, this is instead a test of medians (though not in an exact way as Robert already notes). In the parametric case, this is achieved by apportioning variance related to fixed factors (our groups) and general variance within these groups, such that the $F$ test used in ANOVA is:

$$ F = \frac{MS_{\text{between}}}{MS_{\text{within}}} $$

where $MS$ is the "mean squares". The calculation of the mean squares is quite tedious but can be less formally described as (Gravetter et al., 2021):

$$ F = \frac{ \text{variance between groups} }{ \text{variance within groups} } $$

with the "variance" here being not a test of the actual variance in groups, but variance of sample means. The major difference with the nonparametric Kruskall-Wallis is that testing is done using a rank-based approach instead (as they are less prone to issues with outliers, heterogeneous variance, etc.):

$$ H = \left(\frac{12}{N(N+1)} \sum_{j=1}^k \frac{R_{j}^2}{n_j}\right) - 3(N+1), $$

where $N$ is the total sample size, $n_j$ is the sample size for a given group, $j$ is the group designator, $R_j$ is the sum of the ranks in a group. See this link for a step-by-step approach to calculating the K-W test. The idea behind it is that you are pooling the observations from the samples into one combined sample, then ranking each group observation from lowest to highest from $1$ to $N$, and then comparing the groups after.

ANOVA and PWCs

We can now determine how omnibus tests like these are not making the same claims as their PWCs, which is a big part of your question. Note at the outset we have a potential problem with how we are testing groups in the omnibus form. We are not testing which groups are different from each other (or even how different they are). The test can be heuristically stated as "is there no difference between the groups I have selected?" The ANOVA test, when flagged for statistical significance, simply states that the evidence of no differences is low. As a simple thought experiment, say we collect reading scores from 100 different schools across some country. If we run an ANOVA, we will almost assuredly get flagged for significance. Why? Because at this point, almost any fluctuation in means will enlarge the $F$ ratio. We are also just pooling the results here without determining how this relates to individual groups. To reiterate, we neither find out where our differences lie or how much of a difference actually exists. Hence the use of t-tests, typically corrected pairwise comparisons (PWCs), which largely developed after Fisher's formulation of ANOVA.

I note this specifically because you ask whether or not we should pay attention to one test statistic (the omnibus test) compared to another (the by-group comparisons). It may be useful to explore Midway et al., 2020, which provides a detailed accounting of what PWCs are and which ones are useful to use contingent upon setting. In their paper, they state the following:

The classic ANOVA (ANalysis Of Variance) is a general linear model that has been in use for over 100 years (Fisher, 1918) and is often used when categorical or factor data need to be analyzed. However, an ANOVA will only produce an F -statistic (and associated p-value) for the whole model. In other words, an ANOVA reports whether one or more significant differences among group levels exist, but it does not provide any information about specific group means compared to each other. Additionally, it is possible that group differences exist that ANOVA does not detect. For both of these reasons, a strong and defensible statistical method to compare groups is nearly a requirement for anyone analyzing data.

However, PWCs have their own problems, particularly when you have several group comparisons to make. As ANOVA already has issues with the school example I presented earlier, so too does a typical PWC (usually a sharp reduction in power with many group comparisons, see Nakagawa, 2004). Thankfully you do not have a considerable number of groups, but I note this because you are already using a nonparametric ANOVA and PWCs, which makes the power even further reduced. Whether that is warranted or not is contingent upon your data, but I simply note that it is something to consider with respect to reporting your results.

What This Means for You

First, you can explain the differences in these NHST test statistics separately. For the omnibus test, you can explain that the test that "there exists no difference between the groups" showed that the probability was low, perhaps lower than chance outcomes, so the idea that the differences are exactly zero is perhaps not well-founded. For the PWCs, you simply state that the null hypothesis was/was not rejected for each group. This can be done independent of the omnibus test for the reasons outlined above. See Wasserstein and Lazar (2018) for proper reporting of $p$ values, and perhaps consider neoFisherian reporting (Hurlbert & Lombardi, 2009) so as to not exaggerate your claims.

Second, and more important, never just report $p$ values for testing. They are largely useless on their own, and have a well-documented relationship with causing real-world harm by making poor claims with them (Hauer, 2004; Ziliak & McCloskey, 2008). This is no less true for ANOVAs or PWCs. It is much more important to report things like the raw mean/SD estimates for each group, visualizations of group differences, any effect sizes from both the model and group comparisons, and perhaps things like confidence intervals as well (see Cumming, 2014 for why). I repeat, do not just report the $p$ values, as your post seems to indicate that this was the only thing being considered.

References

  • Nakagawa, S. (2004). A farewell to Bonferroni: The problems of low statistical power and publication bias. Behavioral Ecology, 15(6), 1044–1045. https://doi.org/10.1093/beheco/arh107
  • Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7–29. https://doi.org/10.1177/0956797613504966
  • Fisher, R. A. (1921). Studies in crop variation. I. An examination of the yield of dressed grain from Broadbalk. The Journal of Agricultural Science, 11(2), 107–135. https://doi.org/10.1017/S0021859600003750
  • Hauer, E. (2004). The harm done by tests of significance. Accident Analysis & Prevention, 36(3), 495–500. https://doi.org/10.1016/S0001-4575(03)00036-8
  • Gravetter, F. J., Wallnau, L. B., Forzano, L.-A. B., & Witnauer, J. E. (2021). Essentials of statistics for the behavioral sciences (Edition 10). Cengage.
  • Hurlbert, S. H., & Lombardi, C. M. (2009). Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian. Annales Zoologici Fennici, 46(5), 311–349. https://doi.org/10.5735/086.046.0501
  • Midway, S., Robertson, M., Flinn, S., & Kaller, M. (2020). Comparing multiple comparisons: Practical guidance for choosing the best multiple comparisons test. PeerJ, 8, e10387. https://doi.org/10.7717/peerj.10387
  • Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p -values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108
  • Ziliak, S. T., & McCloskey, D. N. (2008). The cult of statistical significance: How the standard error costs us jobs, justice, and lives. University of Michigan Press.
$\endgroup$
3
  • 2
    $\begingroup$ (+1) This is a fantastic answer, Shawn ! I'm bookmarking it :) $\endgroup$ Nov 20, 2023 at 8:59
  • 1
    $\begingroup$ Thanks! I've been reading a lot about Fisher recently, particularly the differences between him and Neyman-Pearson thinking and the formulation of ANOVA, which largely color my answer here regarding the generalized approach the ANOVA takes. I believe his second paper (Studies in Crop Variation II) is the one that formalizes this approach and actually provides some interesting visualizations of the method for those curious. You are likely aware of it already, but it may be an interesting read to those who haven't. $\endgroup$ Nov 20, 2023 at 9:15
  • 2
    $\begingroup$ @ShawnHemelstrand, you can check Probability Theory and Statistical Inference: Econometric Modeling with Observational Data, by Aris Spanos for the differences between NP and Fisherian paradigms from a historical and philosophical pov. $\endgroup$ Nov 20, 2023 at 10:59
5
$\begingroup$

To me, this question is the same as whether to do any kind of pairwise comparisons after ANOVA. The fact that you used a nonparametric test doesn't change the key issues.

This issue has been discussed here many times e.g. What if an overall ANOVA is not significant but specific contrasts are? which has some good answers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.