I have two small data sets (sizes $n_1 = 8$ and $n_2 = 21$) which look like they have significantly different variances. I know very little about the underlying distributions, but it's definitely not safe to assume they're normal or anything nice like that, which rules out the F-test. I'm aware I could use one of the other named tests (Bartlett, Brown–Forsythe, ...) although I'm not currently quite sure what they assume about the population distribution, if anything.

Instead, I've tried my hand at using a permutation test: the null hypothesis is that the two datasets have equal variance, so relabel the data points at random and measure the difference in the two variances of the relabelled sets. Out of 1,000,000 attempts, fewer than 40,000 had a larger difference in variance (<4%).

Is it correct to say that, therefore, the difference in variance of the two data sets is significant at the $p < 0.05$ level? If so, is there a well-recognised name for this kind of test?

  • $\begingroup$ What do you mean by relabel the data points? I don't get that part $\endgroup$ – machazthegamer May 8 '17 at 9:50
  • 1
    $\begingroup$ It's just a two-sample permutation test with the difference in variance as the test statistic; I don't know of any special name for that (a ratio rather than a difference would perhaps be more typical for variances). Note that there are fewer than 4.3 million combinations; you could almost as easily compute the exact p-values. $\endgroup$ – Glen_b May 8 '17 at 9:50
  • $\begingroup$ @machaz relabel which sample the points are coming from. That's how permutation tests work (at least in the simple cases) $\endgroup$ – Glen_b May 8 '17 at 9:50
  • $\begingroup$ @Glen_b thanks.Now its clear.I hadn't rally used permutation tests before. $\endgroup$ – machazthegamer May 8 '17 at 11:17

Your permutation test setup makes perfect sense - congratulations!

No, as far as I know, there is no established name for this kind of procedure. I'd recommend you describe it as you did here, and potentially refer to any introductory textbook for permutation tests. (You might want to add whether you ran a one-sided or a two-sided test, which is not entirely clear to me from your description. Either one, of course, can be implemented in your permutation test framework.)

  • $\begingroup$ It was a one-sided test. $\endgroup$ – Dave Turner May 8 '17 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.