3
$\begingroup$

I have more than five hundred thousands samples of a continuous variable measured in two groups: a treatment and a control one. I would like to decide whether the measurements follow the same (unknown) distribution in the two groups.

Graphically this seems to be the case: the density functions (obtained using R) for the two groups overlap entirely.

I tried to asses this in a more formal way using the Kolmogorov-Smirnov test. However I obtain a p.value of 2e-13 which suggests that the the two groups do not follow the same distribution.

I am not convinced though and think that this is due to the fact that I have so many measurements that even the slightest difference leads to reject the null hypothesis.

I tried to check if the mean is the same (using Wilcoxon-Manney test) and again the p.value is very low p < 1e-6. However the difference between the mean values in the two groups is quite low (0.006 and the values can range from 0 to 1) which is, for each practical purpose, identical.

Am I using the wrong statistical tests? How can I assess in a formal way whether the two distributions are the same or not?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

Yes, this is a common feature of statistical analyses with very large samples. When your data are observational (this wasn't a true experiment), we generally believe that all parameters will differ at least somewhat, and whether or not a test is significant only depends on N. Although the context differs, you can get the basic idea from this CV thread: Is normality testing 'essentially useless'? What may be more useful for you is equivalence testing. I explain the basic ideas in my answer here: Why do statisticians say a non-significant result means "you can't reject the null" as opposed to accepting the null hypothesis?

Improve this answer
$\endgroup$
4
  • $\begingroup$ thanks :) I will read the topics and get back here if I have other doubts! $\endgroup$
    – lucacerone
    Commented May 8, 2014 at 13:45
  • $\begingroup$ You're welcome, @lucacerone. $\endgroup$
    – gung - Reinstate Monica
    Commented May 8, 2014 at 13:47
  • $\begingroup$ Hi @gung I have two questions: 1. what non parametric tests exist to assess equivalence? I will need to explain why we did not use standard tests, do you have any reference I can use to back up the claim that those are not the most suit tests? $\endgroup$
    – lucacerone
    Commented May 8, 2014 at 14:53
  • $\begingroup$ @lucacerone, in theory you could use whatever test you want to assess equivalence. Equivalence is about what you are testing, not what test you use to do so. Off the top of my head, I'm not quite sure how you'd use the M-W U-test for tost, but I'm sure it can be done. You mostly need to work out the interval that you consider sufficiently close to be considered equivalent. Here is a tutorial publication I found. $\endgroup$
    – gung - Reinstate Monica
    Commented May 8, 2014 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.