3
$\begingroup$

I would like to raise a problem that bothers me. However, I would like to point out right away that my question is purely theoretical, and the data presented here comes only from a computer experiment and was artificially generated.

Suppose I ran an experiment and collected 750 samples of some parameter (x1). Then I changed some input conditions twice and collected the same number of samples twice (x2, x3).

Finally, I would like to check whether the changed input conditions had an impact on the tested parameter (variable).

As a result, I got these results.

# A tibble: 3 x 14
# Groups:   name [3]
  name      n    min     q1     mean median    q3   max    sd kurtosis skewness SW.stat     SW.p  nout
  <fct> <int>  <dbl>  <dbl>    <dbl>  <dbl> <dbl> <dbl> <dbl>    <dbl>    <dbl>   <dbl>    <dbl> <int>
1 x1      750  -7.28 -1.64   0.0233       0 1.76   7.48  2.50   -0.210 -0.00270   0.999 8.39e- 1     4
2 x2      750  -5.49 -0.780  0.00846      0 0.816  5.90  1.36    1.83  -0.00685   0.980 1.76e- 8    21
3 x3      750 -11.1  -0.762 -0.0266       0 0.719 12.6   1.86   15.5    0.482     0.776 8.59e-31    37

As you can see, the median is "absolute zero" in each of the three repetitions. Mean is not much different from zero either. The first and third quartiles are also very similar.

You could say I got exactly the same answer.

Let's see what the ANOVA + t-test will tell you. enter image description here

Yes, I know that this data does not meet the assumptions of the ANOVA test! In that case, let's turn to nonparametric tests. enter image description here

This gives me one answer - in this case there are no statistically significant differences.

However, everyone can see that the answers differ diametrically from each other! If we look at the value of kurtosis, we can see that the kurtosis of the variable x3 is almost 100 times greater than the kurtosis of the variable x1!

And finally my question. But please forgive me, if statistically it is stupid. I am a statistical self-taught. Is there a statistical test that would answer the question of whether there is a statistically significant difference between the kurtosis value?

$\endgroup$
9
  • $\begingroup$ Do you really want to test the kurtosis, or do you just want to show that the three groups are not the same? // Why isn't the graphical assessment enough? $\endgroup$
    – Dave
    Oct 27, 2021 at 16:12
  • 1
    $\begingroup$ You have three groups, but let's pretend you only had two. Would the Kolmogorov-Smirnov test accomplish what you want to do? (It sounds like it to me.) $\endgroup$
    – Dave
    Oct 27, 2021 at 17:01
  • 2
    $\begingroup$ The standard error of the kurtosis is proportional to moments up to order eight! Unless you have millions of data points, it's usually hopeless to estimate the kurtosis with enough accuracy to make a useful test. Indeed, ANOVA does not usually require any kind of formal Normality testing. Searching our site for "Normality testing" or related keywords will turn up many discussions of why this is unnecessary and usually inadvisable. $\endgroup$
    – whuber
    Oct 27, 2021 at 17:25
  • 2
    $\begingroup$ "in this case there are no statistically significant differences." Statistically significant differences in what? $\endgroup$
    – Alexis
    Oct 27, 2021 at 18:21
  • 2
    $\begingroup$ There are tests for differences in variance, like Levene’s or Brown–Forsythe, if that's what one is interested in. ... I think part of the underlying problem is that when we present hypothesis tests like t tests, we use language like, it tests if "there is a difference", or "if the samples come from the same population". ... I wish we would start with, e.g. "this test is sensitive to a difference in means," Or "This test is sensitive to a difference in medians..." $\endgroup$ Oct 31, 2021 at 22:08

2 Answers 2

3
$\begingroup$

A major thing to understand about statistical tests is that they only ever test a certain aspect of the null hypothesis. A standard ANOVA F-test tests equality against the alternative of a difference regarding group means. The F-test is not "interested" in kurtosis (or only because this may affect the power to find mean differences). Similarly Kruskal-Wallis will look for differences in rank sums, which you will find if one group is distributed stochastically larger than another, i.e. has a general tendency to yield larger values. Once more this test is not interested in kurtosis. Kolmogorov-Smirnov (K-S) tests for equality vs. differences in the overall shape of the distribution and will therefore find kurtosis differences. I had originally written that there are also tests for equality of kurtosis in particular, but right now I don't find them in the literature. What surely can be done is the difference between kurtosis values can be used as test statistic in a permutation test.

One may wonder whether not always K-S is applied because it may seem to test for general differences, but because of this (a) it will have a worse power to detect mean differences than standard ANOVA or even Kruskal-Wallis, and (b) in many situations what is of interest are mean differences or rather differences regarding generally delivering larger values for one group than another, but not differences in kurtosis (e.g., "is one treatment better than another?").

PS: A comment mentions Mahmoudi, Mohammad Reza, Bui Anh Tuan, and Kim-Hung Pho. 2021. “On Kurtoses of Two Symmetric or Asymmetric Populations.” Journal of Computational and Applied Mathematics 391 (August): 113370. doi.org/10.1016/j.cam.2020.113370.

$\endgroup$
5
  • $\begingroup$ Christian wrote "there are also tests for equality of kurtosis in particular". Then what test should I use. In all this consideration, I was concerned only with determining whether there is a difference between x1 and x3 and whether it is statistically significant. $\endgroup$ Oct 31, 2021 at 21:32
  • $\begingroup$ If you are interested in any difference, K-S is fine. $\endgroup$ Oct 31, 2021 at 21:34
  • $\begingroup$ Ok. But is this kurtosis test a mystery among statisticians? You can tell me what this test is. As I wrote at the beginning, I am a statistical self-taught person. $\endgroup$ Oct 31, 2021 at 21:36
  • $\begingroup$ @MarekFiołka Interesting... I thought I had seen a test comparing the kurtosis of two samples but don't find it now. What certainly can be done is the difference between the two sample kurtosis values can be used as a test statistic, and the test can be run as permutation test. Not sure though whether this is somewhere in the literature. It's surely not a standard thing. $\endgroup$ Oct 31, 2021 at 21:46
  • 1
    $\begingroup$ I found a recent article for testing the ratio of two kurtosis coefficients Mahmoudi, Mohammad Reza, Bui Anh Tuan, and Kim-Hung Pho. 2021. “On Kurtoses of Two Symmetric or Asymmetric Populations.” Journal of Computational and Applied Mathematics 391 (August): 113370. doi.org/10.1016/j.cam.2020.113370. $\endgroup$
    – Josef
    May 22, 2022 at 14:14
-1
$\begingroup$

@Dave you were right! Here the Kolmogorov-Smirnov test should have been applied! It is a pity that you did not provide this in the form of an answer, but only a comment. However, I decided to answer my question myself, because maybe someone will have a similar problem someday.

Below is the same graph as in my question, but now I have used the Kolmogorov-Smirnov test as a pair-wise test. enter image description here

A small update for @Alexis

Dear @Alexis, you ask if I understand this. Well, I must admit, not too much. As I wrote, I am self-taught. Moreover, English is not my mother tongue. So I am apprehensive that what I am writing looks like gibberish.

Let me write what I understand. First, I look at the classic box-plot.

enter image description here

What do I see here? I can see the medians are almost identical. I can see that the averages must be very close to the medians and they should also be very similar.

I can see that x1 has a slightly larger interquartile range and x3 has a larger range of value.

I also see quite significant differences in Tukey fences.

So my first conclusion is that there is something different.

Then I look at the numbers.

  name      n    min     q1     mean median    q3   max    sd kurtosis skewness SW.stat     SW.p  nout
  <chr> <int>  <dbl>  <dbl>    <dbl>  <dbl> <dbl> <dbl> <dbl>    <dbl>    <dbl>   <dbl>    <dbl> <int>
1 x1      750  -7.28 -1.64   0.0233       0 1.76   7.48  2.50   -0.210 -0.00270   0.999 8.39e- 1     4
2 x2      750  -5.49 -0.780  0.00846      0 0.816  5.90  1.36    1.83  -0.00685   0.980 1.76e- 8    21
3 x3      750 -11.1  -0.762 -0.0266       0 0.719 12.6   1.86   15.5    0.482     0.776 8.59e-31    37

Again, I see that both the medians and the mean are not different at all. I also confirm my observations regarding the range of values and the interquartile range.

I also see that only about the x1 samples I can say that they come from the normal distribution.

I also see a wide variety of kurtosis and the number of outliers.

And now I ask myself. Could the differences I perceive be coincidental? Is there any test tool that I could prove with the assumed level of confidence that these differences are statistically real? Will I even ask meaningful questions?

If you have answers to any of the questions that bother me, please answer them. The overriding idea on a CV is that we all learn from each other. And I am very eager to learn something.

$\endgroup$
7
  • 2
    $\begingroup$ This "answer" provides no indication that you understand differences in what? $\endgroup$
    – Alexis
    Oct 31, 2021 at 21:59
  • $\begingroup$ Dear @Alexis I added a small update to my question-answer. $\endgroup$ Nov 3, 2021 at 19:36
  • 1
    $\begingroup$ Marek Fiolka: Statistical tests do not provide evidence for some mysteriously unspecified "difference," but for quite concretely articulated null and alternative hypotheses, which are about differences in population means, ratios of variances, relative stochastic size, etc.. $\endgroup$
    – Alexis
    Nov 3, 2021 at 22:27
  • $\begingroup$ Dear @Alexis, forgive me, but I don't understand what you are writing. Suppose I am an engineer overseeing a certain parameter in a process. Every time I take samples, I get something that is very similar to x1. However, I decided to make a few improvements. As a result, I got x2 and x3. Please explain to me what is wrong if I am wondering if my modification affected the results in any way? What's wrong with trying to find the difference? What's wrong with the idea @Dave suggested? What is wrong with my answer which was the realization of his idea? $\endgroup$ Nov 6, 2021 at 20:11
  • $\begingroup$ What is wrong with checking (since there are no differences between means and ranks) that x1 and x2 come from the same distributions? Did I show the results of the Kolmogorov-Smirnov test incorrectly by writing the p-value on the graph? Or maybe you mean it that the rstatix package does not include the pairwise_ks_test function? Well, I admit I did it myself. But is it really bad? And if you think you know a better method or path, I'd like to reiterate that you have the opportunity to post your answer here. $\endgroup$ Nov 6, 2021 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.