I'm trying to compare a number of large data sets (over 3000 samples per set), when I plot histograms of these distributions I can see two which are clearly different and a z-test confirms this, which is great.

However, there are two distributions which look so alike I couldn't tell them apart without knowing which was which. A z-test says these distributions are also significantly different. I've heard that the z-test breaks down a bit with huge samples so I'm guessing this significance is due to some minor difference that is not realistic.

What I want to know is if there is an alternative test for large data sets, or whether the z test can be modified for this?

  • 2
    $\begingroup$ The z-test doesn't "break down" with large samples, it behaves exactly as it is supposed to. The problem is you're using hypothesis tests when they don't correspond to what you actually want to do. If you insist on whacking screws with a hammer, it's not the hammer's fault your cabinets look funny and the doors occasionally fall off. The fact that you only notice the problem with very long screws doesn't alter the fact that it's the wrong tool for the job even when the screw is short. Rather than "what should I do different in large samples", focus on answering the right question at any $n$. $\endgroup$
    – Glen_b
    Sep 4 '14 at 2:41
  • $\begingroup$ If you're "looking" at multiple histograms and then select a subset of these to submit them to z-tests, out of the many, many z-tests you could be doing, you're violating an independence assumption and should correct for multiple tests (for the tests you could have done, but didn't, because you saw the plots). $\endgroup$
    – jona
    Sep 4 '14 at 21:39

There is no law against not using a statistical test at all when the effect you see is not meaningful to you. The purpose of statistical tests is to see if a result that could be important is real, or due to chance. That isn't the character of what you need to do here.

I'll add that if it's really important for you to do a test, you could also test for equivalence: set up a threshold value, $\delta$ whereby any difference less than $\delta$ is deemed unimportant. Then do two one-sided $z$ tests -- one of $H_0^1: \mu_1-\mu_2\ge-\delta$ and the other of $H_0^2: \mu_1-\mu_2\le\delta$. If you reject both hypotheses, you'll have strong evidence that $-\delta<\mu_1-\mu_2\ <\delta$. The easy way to do this is to calculate a $(1-2\alpha)$ confidence interval for $\mu_1-\mu_2$, and reject both hypotheses if the interval lies entirely within $(-\delta,\delta)$.

It is possible - especially with large datasets - to have both a significant difference and significant equivalence, so you need to be prepared to explain why that can happen.

Addendum: Testing for "supersignificance"

Yet another idea -- and this may be closer to answering the original question -- is to turn that equivalence the other way, and test $$ H_0^\delta: |\mu_1-\mu_2|\le\delta \qquad \mbox{vs.} \qquad H_1^\delta: |\mu_1-\mu_2|>\delta $$ This is saying that a difference of $\delta$ or larger is required to be considered important. This can be tested via the test statistic: $$ z(\delta) = \frac{|\bar y_1 - \bar y_2| - \delta}{\sqrt{s_1^2/n_1 + s_2^2/n_2}} $$ The decision rule is to reject $H_0^\delta$ in favor of $H_1^\delta$ if $z(\delta) > z_{\alpha/2}$ (e.g., $z(\delta)>1.96$ if using $\alpha=.05$). Note that if we choose $\delta=0$, we have $z(0)=|z|$, the absolute value of the usual $z$ statistic, and this is then just the usual $z$ test.

Interestingly, the same statistic can be used to test for equivalence, for which the null hypothesis is $H_1^\delta$ and the alternative is $H_0^\delta$ (!!!). Specifically, we reject $H_1^\delta$ in favor of $H_0^\delta$ if $z(\delta) < -z_\alpha$ (e.g., $z(\delta)<-1.645$ with $\alpha=.05$). If $z(\delta)$ is between these two critical values, we have an inconclusive result, and clearly it is not possible for both to be significant. Note also that $z(0) \not< 0$, so you can't conclude equivalence with a threshold of $\delta=0$.

  • $\begingroup$ Does anybody know if there's a commonly used name for this "supersignificance" test? I sure didn't find any relevant Google hits with that term. $\endgroup$
    – Russ Lenth
    Sep 4 '14 at 17:06
  • 1
    $\begingroup$ The null hypothesis is called (by people like Jacob Cohen) a "nil" hypothesis, and he has talked about using a different value, but I don't recall it having a name. (And "non-nil null hypothesis" is too hard to say.) $\endgroup$ Sep 4 '14 at 21:36
  • $\begingroup$ You got my +1 with the first sentence. $\endgroup$
    – whuber
    Sep 4 '14 at 21:41
  • 1
    $\begingroup$ @RussLenth The common name for combining inference from tests for difference (H$^{+}_{0}$) with tests for equivalence (H$^{–}_{0}$) is "relevance testing" (See my answer here: stats.stackexchange.com/questions/108911/…). See there also: the interpretation of the rejection of both H$^{+}_{0}$ and H$^{–}_{0}$ implies trivial difference: the difference is significant, but you have a priori said you don't care about differences that small. $\endgroup$
    – Alexis
    Sep 4 '14 at 21:43

If you are comparing distributions by their shape a more relevant test would be the two-sample K-S test, particularly given the large number of observations. A K-S test is a comparison of the empirical CDF's (http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test).

Two sided K-S tests are available in R through "ks.test".

  • $\begingroup$ I agree with this. My answer focuses on comparing means, but with the way the question is stated, and the fact that you definitely have enough data to do this, this would be a more appropriate way to go. And you could decide a priori how large a difference between CDFs is enough to be relevant to you (suggestion: $\delta_F=\max_x\{\Phi((x-\delta)/\sigma) - \Phi(x/\sigma)\}$ where $\delta$ is a minimally important shift in means (like in my answer) and $\Phi$ is the normal CDF (or some other standardized CDF). $\endgroup$
    – Russ Lenth
    Sep 4 '14 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.