4
$\begingroup$

I have two small data sets (sizes $n_1 = 8$ and $n_2 = 21$) which look like they have significantly different variances. I know very little about the underlying distributions, but it's definitely not safe to assume they're normal or anything nice like that, which rules out the F-test. I'm aware I could use one of the other named tests (Bartlett, Brown–Forsythe, ...) although I'm not currently quite sure what they assume about the population distribution, if anything.

Instead, I've tried my hand at using a permutation test: the null hypothesis is that the two datasets have equal variance, so relabel the data points at random and measure the difference in the two variances of the relabelled sets. Out of 1,000,000 attempts, fewer than 40,000 had a larger difference in variance (<4%).

Is it correct to say that, therefore, the difference in variance of the two data sets is significant at the $p < 0.05$ level? If so, is there a well-recognised name for this kind of test?

$\endgroup$
4
  • $\begingroup$ What do you mean by relabel the data points? I don't get that part $\endgroup$ – machazthegamer May 8 '17 at 9:50
  • 1
    $\begingroup$ It's just a two-sample permutation test with the difference in variance as the test statistic; I don't know of any special name for that (a ratio rather than a difference would perhaps be more typical for variances). Note that there are fewer than 4.3 million combinations; you could almost as easily compute the exact p-values. $\endgroup$ – Glen_b May 8 '17 at 9:50
  • $\begingroup$ @machaz relabel which sample the points are coming from. That's how permutation tests work (at least in the simple cases) $\endgroup$ – Glen_b May 8 '17 at 9:50
  • $\begingroup$ @Glen_b thanks.Now its clear.I hadn't rally used permutation tests before. $\endgroup$ – machazthegamer May 8 '17 at 11:17
2
$\begingroup$

Your permutation test setup makes perfect sense - congratulations!

No, as far as I know, there is no established name for this kind of procedure. I'd recommend you describe it as you did here, and potentially refer to any introductory textbook for permutation tests. (You might want to add whether you ran a one-sided or a two-sided test, which is not entirely clear to me from your description. Either one, of course, can be implemented in your permutation test framework.)

$\endgroup$
6
  • $\begingroup$ It was a one-sided test. $\endgroup$ – Dave Turner May 8 '17 at 14:13
  • 1
    $\begingroup$ There is an established name for such a procedure: it's called the Levene test for homoegeneity of variances. $\endgroup$ – whuber Aug 25 '20 at 14:16
  • 1
    $\begingroup$ @whuber: thanks, that's good to know. I even used Levene's test a couple of years ago, but apparently only the idea stuck... $\endgroup$ – Stephan Kolassa Aug 25 '20 at 14:31
  • 2
    $\begingroup$ @COOL I wasn't trying to claim Levene's test is a permutation test. The point is that this question ultimately asks for nonparametric tests of variances and that's what Levene's is. Actually, your inquiry does reflect a deeper insight: many (maybe most?) nonparametric tests based on ranks indeed are equivalent to permutation tests. That is because the null hypotheses usually amount to supposing all possible permutations are equally likely. $\endgroup$ – whuber Aug 25 '20 at 16:14
  • 1
    $\begingroup$ @whuber Ah got it, thanks for the explanations. $\endgroup$ – COOLSerdash Aug 25 '20 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.