Statistical significance test of kurtosis differences

Question

I would like to raise a problem that bothers me. However, I would like to point out right away that my question is purely theoretical, and the data presented here comes only from a computer experiment and was artificially generated.

Suppose I ran an experiment and collected 750 samples of some parameter (x1). Then I changed some input conditions twice and collected the same number of samples twice (x2, x3).

Finally, I would like to check whether the changed input conditions had an impact on the tested parameter (variable).

As a result, I got these results.

# A tibble: 3 x 14
# Groups:   name [3]
  name      n    min     q1     mean median    q3   max    sd kurtosis skewness SW.stat     SW.p  nout
  <fct> <int>  <dbl>  <dbl>    <dbl>  <dbl> <dbl> <dbl> <dbl>    <dbl>    <dbl>   <dbl>    <dbl> <int>
1 x1      750  -7.28 -1.64   0.0233       0 1.76   7.48  2.50   -0.210 -0.00270   0.999 8.39e- 1     4
2 x2      750  -5.49 -0.780  0.00846      0 0.816  5.90  1.36    1.83  -0.00685   0.980 1.76e- 8    21
3 x3      750 -11.1  -0.762 -0.0266       0 0.719 12.6   1.86   15.5    0.482     0.776 8.59e-31    37

As you can see, the median is "absolute zero" in each of the three repetitions. Mean is not much different from zero either. The first and third quartiles are also very similar.

You could say I got exactly the same answer.

Let's see what the ANOVA + t-test will tell you.

Yes, I know that this data does not meet the assumptions of the ANOVA test! In that case, let's turn to nonparametric tests.

This gives me one answer - in this case there are no statistically significant differences.

However, everyone can see that the answers differ diametrically from each other! If we look at the value of kurtosis, we can see that the kurtosis of the variable x3 is almost 100 times greater than the kurtosis of the variable x1!

And finally my question. But please forgive me, if statistically it is stupid. I am a statistical self-taught. Is there a statistical test that would answer the question of whether there is a statistically significant difference between the kurtosis value?

Answer 1

A major thing to understand about statistical tests is that they only ever test a certain aspect of the null hypothesis. A standard ANOVA F-test tests equality against the alternative of a difference regarding group means. The F-test is not "interested" in kurtosis (or only because this may affect the power to find mean differences). Similarly Kruskal-Wallis will look for differences in rank sums, which you will find if one group is distributed stochastically larger than another, i.e. has a general tendency to yield larger values. Once more this test is not interested in kurtosis. Kolmogorov-Smirnov (K-S) tests for equality vs. differences in the overall shape of the distribution and will therefore find kurtosis differences. I had originally written that there are also tests for equality of kurtosis in particular, but right now I don't find them in the literature. What surely can be done is the difference between kurtosis values can be used as test statistic in a permutation test.

One may wonder whether not always K-S is applied because it may seem to test for general differences, but because of this (a) it will have a worse power to detect mean differences than standard ANOVA or even Kruskal-Wallis, and (b) in many situations what is of interest are mean differences or rather differences regarding generally delivering larger values for one group than another, but not differences in kurtosis (e.g., "is one treatment better than another?").

PS: A comment mentions Mahmoudi, Mohammad Reza, Bui Anh Tuan, and Kim-Hung Pho. 2021. “On Kurtoses of Two Symmetric or Asymmetric Populations.” Journal of Computational and Applied Mathematics 391 (August): 113370. doi.org/10.1016/j.cam.2020.113370.

Answer 2

@Dave you were right! Here the Kolmogorov-Smirnov test should have been applied! It is a pity that you did not provide this in the form of an answer, but only a comment. However, I decided to answer my question myself, because maybe someone will have a similar problem someday.

Below is the same graph as in my question, but now I have used the Kolmogorov-Smirnov test as a pair-wise test.

A small update for @Alexis

Dear @Alexis, you ask if I understand this. Well, I must admit, not too much. As I wrote, I am self-taught. Moreover, English is not my mother tongue. So I am apprehensive that what I am writing looks like gibberish.

Let me write what I understand. First, I look at the classic box-plot.

What do I see here? I can see the medians are almost identical. I can see that the averages must be very close to the medians and they should also be very similar.

I can see that x1 has a slightly larger interquartile range and x3 has a larger range of value.

I also see quite significant differences in Tukey fences.

So my first conclusion is that there is something different.

Then I look at the numbers.

  name      n    min     q1     mean median    q3   max    sd kurtosis skewness SW.stat     SW.p  nout
  <chr> <int>  <dbl>  <dbl>    <dbl>  <dbl> <dbl> <dbl> <dbl>    <dbl>    <dbl>   <dbl>    <dbl> <int>
1 x1      750  -7.28 -1.64   0.0233       0 1.76   7.48  2.50   -0.210 -0.00270   0.999 8.39e- 1     4
2 x2      750  -5.49 -0.780  0.00846      0 0.816  5.90  1.36    1.83  -0.00685   0.980 1.76e- 8    21
3 x3      750 -11.1  -0.762 -0.0266       0 0.719 12.6   1.86   15.5    0.482     0.776 8.59e-31    37

Again, I see that both the medians and the mean are not different at all. I also confirm my observations regarding the range of values and the interquartile range.

I also see that only about the x1 samples I can say that they come from the normal distribution.

I also see a wide variety of kurtosis and the number of outliers.

And now I ask myself. Could the differences I perceive be coincidental? Is there any test tool that I could prove with the assumed level of confidence that these differences are statistically real? Will I even ask meaningful questions?

If you have answers to any of the questions that bother me, please answer them. The overriding idea on a CV is that we all learn from each other. And I am very eager to learn something.