I have a fundamental question: is it reasonable to use a statistical test to find out if 2 datasets are similar or not? Some comments have called out the T-test as being unable to answer that question. Why is the T-test insufficient? Furthermore, what other tests are there to determine differences in distributions for the two sample problem.
$\begingroup$
$\endgroup$
3
-
$\begingroup$ This question has a problem: although it makes perfect sense, it really does not become clear enough to answer until you stipulate more precisely what you mean by "similar." Some people are satisfied when two datasets are centered near the same values; others would like them to have similar histogram shapes ("distributions"); in yet other applications only the highest or lowest values need to be approximately the same. The scope of such possibilities ought to alert you to how broad and general your question is, too: please consider editing your question to clarify and focus it. $\endgroup$– whuber ♦Commented May 6, 2014 at 16:47
-
$\begingroup$ Concerning "similar" i have meant the same distribution. But would be a histogram enough? I always thought that a t-test would show that 2 datasets are "similar" or not with regarding the coincidence. What if the distribution of both datasets is only similar because of a coincidence? Wouldnt a t-test help to find this out? $\endgroup$– mister nobodyCommented May 6, 2014 at 17:05
-
$\begingroup$ A t-test only compares the means of the distributions (of which the two datasets are assumed to be random samples). This is why I found your question confusing: obviously comparing only the means is not at all the same as comparing the distributions. Thus it seems that your question is put very well in the title and the last line, but the first three lines are irrelevant and distracting. $\endgroup$– whuber ♦Commented May 6, 2014 at 17:09
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
I have a fundamental question: is it reasonable to use a statistical test to find out if 2 datasets are similar or not?
Some comments have called out the T-test as being unable to answer that question. Why is the T-test insufficient?
Two issues:
1) A data set may have very similar mean to a second one, yet substantively differ from the other in other ways (different spread, different shape)
2) The usual hypothesis tests don't tell you if the data sets are necessarily "similar"; they may detect if they're more different than could be explained by random variation. You may be thinking of something more in the area of equivalence testing. Alternatively it may be that your underlying question is more closely related to looking at an effect size than hypothesis testing.
Furthermore, what other tests are there to determine differences in distributions for the two sample problem.
There are tests for many different aspects of distributions, it depends on what you're interested in. People may use a Levene test or perhaps Browne-Forsythe test to compare spreads, or if we look toward nonparametric test, a Siegel-Tukey perhaps an Ansari-Bradley test.
To compare the entire distribution, there are two-sample Kolmogorov-Smirnov tests.
[Note that hypothesis tests are really for making inferences about populations on the basis of samples.]
$\begingroup$
$\endgroup$
8
To be formal about it, there are very abstract non-parametric tests of whether or not the probability distributions of two samples are equal or not. A test that is powered to detect differences in samples under any conditions is the Kullback-Leibler test of divergence. It is a test of the two-sample strong null hypothesis,
$$\mathcal{H}_0: \mathcal{F}_0 = \mathcal{F}_1$$
Note there's no parameter here, like a mean. But what can we say about tests that are powered to detect an infinite number of differences under all possible conditions? They're very rarely practical. The mean is a useful summary statistic for virtually all probability distributions: uniform, beta, weibull, gamma, etc. If there is a useful difference between two distributions, usually there is a difference in means.
-
1$\begingroup$ It is interesting that in some answers, and in many comment threads on this site, people have argued that certain non-parametric distribution tests (such as the Kolmogorov Smirnov) are too powerful. (I don't think anyone has yet argued, though, that such tests are "rarely practical.") This appears to be just about the opposite of the conclusion you draw. $\endgroup$– whuber ♦Commented May 7, 2014 at 14:09
-
$\begingroup$ Thx AdamO, that is what i have been looking for. I have also found a suitable test so called "Kolmogorow-Smirnow-Test". What do you think about this test? $\endgroup$ Commented May 7, 2014 at 14:29
-
$\begingroup$ @whuber earlier you alluded to the problem that the question, as stated, is too vague. I agree. I think the KS test is perfectly suited to assess this vague question. I disagree with the test and the question for any practical application. If I were tasked with this analysis, I would decide upon a few key measurements to compare distributions, rather than select a test which is powered to detect a difference at the 99.999th quantile of two distributions. $\endgroup$– AdamOCommented May 7, 2014 at 15:02
-
$\begingroup$ @user3341057 yes the KS is a test based on the KL divergence. I hope that my and whuber's comments on this answer and on your question can convince you NOT to use this test, however. $\endgroup$– AdamOCommented May 7, 2014 at 15:04
-
$\begingroup$ Adam, it is not at all clear to me that the KS test should be avoided in this application. It certainly has good uses and one of them, when the distribution of the KS test statistic is correctly computed, is to compare two empirical distributions. On a different note, could you enlighten me on how the KS statistic is related to a KL divergence? I do not see any connection: they appear to be completely different things. $\endgroup$– whuber ♦Commented May 7, 2014 at 15:28
$\begingroup$
$\endgroup$
A t-test, like any hypothesis test, can show strong evidence that something is not like it should be. If one fails to prove than, one can't say that the opposite is true. The framework of hypothesis test consider some state as "the natural" state. This state is assumed if there are no evidence against it and cannot be proved.
