Consider $X\sim N(1,1)$ and $Y\sim exp(1)$. The distributions are totally different, but they do have equal variance, so if I were to sample from these and test for unequal variance, I would not want to reject a null hypothesis of variance equality.

I am finding that the usual suspects for variance testing reject too often. In my Monte Carlo simulations, the F-test (var.test in R), Brown-Forsythe (car::leveneTest(…, center="median")), Levene (car::leveneTest(…, center="mean")), Ansari-Bradley (ansari.test), and permutation testing (my own custom function) all rejected considerably more often that my chosen $\alpha$. These tests are overpowered for this situation. I suspect (particularly for Ansari-Bradley) that this is due to the tests examining some kind of surrogate for variance (some kind of "scale" parameter) rather than variance itself, and they are finding that $X$ and $Y$ do differ on that surrogate.

What would be a test that has appropriate power for my $X$ and $Y?$

  • $\begingroup$ The choice of test ought to depend on the amount of data and on any additional assumptions you can make about the distributions. Could you provide some specifics of your intended application? $\endgroup$ – whuber Dec 1 '20 at 15:12
  • $\begingroup$ For amount of data, I won’t be working with 3-4 observations. I would expect (usually) at least dozens of not hundreds of points. For intended application, I have bizarre empirical distributions that don’t seem to follow the named distributions. Mostly, I’m content to use something like Ansari-Bradley (a collaborator adores it), but I’d sure like to see something that really examines variance and not the “scale” that Ansari-Bradley examines, since that results in a false rejection of variance equality for $N(1,1)$ and $exp(1)$. $\endgroup$ – Dave Dec 1 '20 at 15:29
  • $\begingroup$ For comparing variances you typically need much more data than for comparing means. In this light, even a few hundred values is not a large dataset. The A-B test is a rank sum test, which means it may be appropriate for your broad nonparametric assumption: but it also implies it may lack power. The power can be regained to the extent you can narrow your characterization of the distributions your are comparing. I don't follow your distinction between a scale parameter and variance, because the two are interchangeable in any scale family (which this is). $\endgroup$ – whuber Dec 1 '20 at 15:51
  • $\begingroup$ The normal $X$ and exponential $Y$ are in the same scale family? $\endgroup$ – Dave Dec 1 '20 at 16:01
  • $\begingroup$ Yes, because you have implicitly formulated this as a nonparametric problem. If that's not what you mean--if instead you literally mean you know you have sampled from this Normal and this Exponential distribution--then there's nothing left to say. In fact, you don't even need a sample to determine whether these distributions have equal variances! $\endgroup$ – whuber Dec 1 '20 at 16:06

In this case the means of the two distributions are equal. So instead of using a variance test, you can test the hypothesis $E[X^2]=E[Y^2]$.

  • $\begingroup$ This is an interesting idea, but it turns out to give too many type I errors, even when I subtract each variable’s mean from itself to work with zero-mean empirical distributions. $\endgroup$ – Dave Dec 1 '20 at 13:00
  • $\begingroup$ The OP hasn't suggested they would assume the means are equal. Regardless, how would you test equality of the expected squares? Remember, one explicit assumption is that "the distributions are totally different," so unless there is a huge amount of data, applying a standard t-test would be questionable. $\endgroup$ – whuber Dec 1 '20 at 15:11
  • $\begingroup$ Wilcoxon's test is designed for this. $\endgroup$ – chrishmorris Dec 2 '20 at 9:41
  • $\begingroup$ Could you please explain how Wilcoxon's test handles this? Just square the values and throw them in wilcox.test in R? $\endgroup$ – Dave Dec 23 '20 at 20:59
  • $\begingroup$ yes, pass in the two sets of (squared) values, with paired=False $\endgroup$ – chrishmorris Jan 1 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.