8
$\begingroup$

Suppose I have two datasets, $\mathbf{a}$ and $\mathbf{b}$. I want to test whether the two datasets are different in a statistically significant way.

To compute the F-test, I take the ratio of the variances of each dataset and compare this to F values based on some significance level (e.g. $\alpha = 0.05$) and the number of degrees of freedom. If the F value I computed lies outside the bounds of $1\pm\alpha$, then the null hypothesis is rejected (i.e. the two datasets are different in a statistically significant way).

To compute the KS test, I find the ECDF of each dataset and the find the maximum vertical distance between the ECDFs to compute the D-statistic. Similar, to the F-test, if the D-statistic is greater than some critical value, the null hypothesis is rejected (i.e. the two datasets are different in a statistically significant way).

My intuition is that the tests should generally give similar results. If something is statistically significant, it should be statistically significant for both tests, no? Perhaps this intuition is wrong. But, at the very least, I thought that the KS test was more sensitive than the F-test. As such, if the F-test rejects the null hypothesis, then I thought for sure, the KS test would also reject the null.

But I have found many cases where this is not true. I have some examples where the F-test results in rejection of the null hypothesis while the KS test does not!

Any explanation of why this could be is appreciated.

Improve this question
$\endgroup$
4
  • 4
    $\begingroup$ Because they have two different null hypotheses and are therefore testing different things? $\endgroup$
    – Alexis
    Dec 31, 2019 at 20:24
  • $\begingroup$ Even if they were to have had the same null (imagine the question was asking about two tests of goodness of fit of normality, say), they have non-equivalent test statistics (there's a different partial order imposed on the sample space) -- they would measure sample deviation from the null differently. As a result you should typically see differences in p-values. $\endgroup$
    – Glen_b
    Jan 1, 2020 at 1:53
  • $\begingroup$ While it has been hinted at in the above comments, note that the $F$-test requires normality, while the KS test does not. (From another point of view, the $F$-test’s null hypothesis includes that your samples are from normally distributed populations.) The question is still valid if there were no such assumption in the $F$-test, but beware that your examples may simply arise from the $F$-test “detecting” non-normality. $\endgroup$
    – Wrzlprmft
    Jan 1, 2020 at 13:47
  • $\begingroup$ I ran a quick simulation, comparing random samples, normal distribution, n of 25, mean of 0, sd of either 1 or 2. In this case, R's var.test reported p <= 0.05, 92% of the time. And R's ks.test reported p <= 0.05, 15% of the time. So at least under the conditions of this simulation, the KS test is severely underpowered relative to the F test to detect a difference in variance (unless I did something wrong). $\endgroup$
    – Sal Mangiafico
    Jan 1, 2020 at 13:48

2 Answers 2

Reset to default
9
$\begingroup$

Significance testing consists of defining a rejection region, and rejecting if the data is in that region. The size of the region is its $\alpha$ value. If two different regions are different shapes, then even if one is smaller than the other, there can be places that are inside the smaller one but not in the larger one.

Dave’s answer explains that KS tests many different attributes, such as mean, variance, and multimodality. Suppose we restrict our attention to just mean and variance. We can then represent the sample on a two-dimensional plot, with one, say, differences in mean being the horizontal dimension and difference in variance being vertical:

Illustration of rejection regions

The $F$-test’s rejection region (blue) are two horizontal strips in this space: if difference in variance is too positive, or too negative, it rejects the null. The KS test’s rejection region (green) is (with some simplification) a ring: anything too far from the origin in any direction will be rejected. We can (again, with some simplification), consider each to have a “radius”, and anything outside that radius results in the null being rejected. But for the $F$-test, only the vertical distance from the $x$-axis is considered, while the distance from the origin is considered for the KS test.

If both have the same $\alpha$, then since the KS looks at both dimensions, its radius has to be larger. So if your sample has a small difference in mean, and a difference in variance that is slightly more than the $F$-test’s “radius”, then it will be within the KS radius.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Indeed, you can draw such a plot, simply by assuming a specific distribution and simulating a bunch (an obvious candidate for the distribution to begin with would be the normal, since that lets the F have the right type I error rate). You can start with say a standard normal for one group and see how varying the mean and variance of the other affects the rejection rate at some given pair of sample sizes (the specific location and scale you start at doesn't matter, you can generalize to other choices). $\endgroup$
    – Glen_b
    Jan 1, 2020 at 22:59
  • $\begingroup$ It would be somewhat akin to the work required to generate this plot, in that case comparing non-rejection regions for a three group two-way ANOVA and the corresponding Kruskal-Wallis test (two parameters of interest). This is discussed here $\endgroup$
    – Glen_b
    Jan 2, 2020 at 0:43
  • $\begingroup$ Nice. I have not seen this kind of graphical explanation before. $\endgroup$
    – Alexis
    Jan 4, 2020 at 18:56
  • $\begingroup$ Wow that is a great figure to explain the problem. Where did you get it from so I can cite it properly if I use it? $\endgroup$
    – Darcy
    Apr 29, 2020 at 19:27
10
$\begingroup$

The F-test specifically examines for differences in variance and does not need to be sensitive to other differences such as mean. KS has to be sensitive to every kind of distributional difference, whether that difference is mean, variance, or multimodality.

Think of the F-test as a specialist that will be great at finding differences in variance at the expense of perhaps missing other differences. If all you want is to check for variance differences, use the test that specializes in variance differences. If your question is more broad, if there is any difference between the populations, then KS would be the more appropriate test. A drawback to KS is that it won’t tell you what that difference is, while an F test would signal a difference in variances (as a t-test would signal a difference in means).

You are correct that if your populations have different variances and F finds that while KS misses it, that is a type II error by KS.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Specifically the F test's differences in within group and between group variances. $\endgroup$
    – Alexis
    Dec 31, 2019 at 20:25
  • 3
    $\begingroup$ @Alexis ANOVA is one use of the F-test, but nothing prevents us from using F to examine for variance differences between a control and treatment group. $\endgroup$
    – Dave
    Dec 31, 2019 at 20:30
  • 1
    $\begingroup$ The gentle reminder is appreciated, thank you. $\endgroup$
    – Alexis
    Jan 1, 2020 at 0:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.