2
$\begingroup$

I want to compare two data samples which come from continuous distributions. However, the data is highly skewed and t-test is obviously not a choice. As an alternative, I wanted to use the MWW test for locational shift, but one of the assumptions is that the distributions of the groups should be similar, but in my case one group has a much longer tail than the other.

My last choice was the KS test used to compare arbitrary distributions. However, after running the test, the result was not significant, and so the conclusion is that there isn't enough evidence that the distributions of the two groups differ significantly.

What troubles me is that I argued not to use MWW because the distributions differ, but KS said they don't differ significantly. Can somebody explain if my reasoning is valid, or is there something I forgot to consider?

Edit: this is how the distributions look like Boxplots of the groups

ECDFs of the groups

$\endgroup$
11
  • $\begingroup$ "but one of the assumptions is that the distributions of the groups should be similar" - where's that from? Not really. (Although there are some specific situations in which WMW can be misleading.) How many observations do you have in your two groups? Can you show the distributions? Of course we can't know otherwise how reliable your "much longer tail" assessment is. $\endgroup$ Jun 15, 2023 at 22:58
  • $\begingroup$ "What troubles me is that I argued not to use MWW because the distributions differ, but KS said they don't differ significantly. " Yeah, that should trouble you. Maybe your intuition for spotting differences is wrong. $\endgroup$ Jun 15, 2023 at 22:59
  • $\begingroup$ Note that WMW will reject if one distribution produces systematically bigger values than the other, whereas KS looks for any kind of difference, not necessarily "one generally bigger than the other". Depending on what exactly your data mean and what your aim is, one of these may be more appropriate than the other. $\endgroup$ Jun 15, 2023 at 23:02
  • $\begingroup$ @ChristianHennig, I have added the example of the distributions of the groups, both groups have over 100 observations. The assumption of the similarity of distributions was my conclusion since MWW tests the locational shift, and since one group is more right skewed than the other (and also more spread), they can't just differ in the shift. $\endgroup$
    – m_delx
    Jun 15, 2023 at 23:27
  • $\begingroup$ Looking at the boxplots, assuming that the numbers of observations in both groups are very similar, I find it rather astonishing that KS doesn't give you a significant result there. The first thing I'd do is check that it was done correctly. $\endgroup$ Jun 16, 2023 at 9:13

1 Answer 1

2
$\begingroup$

[I apologize in advance for the tone of exasperation that comes into what I write below when the question suddenly shifts from appearing to ask for advice about what test to do to revealing that you already tested the hypothesis. I'll leave it as is, but be prepared for that.]

I want to compare two data samples which come from continuous distributions.

This is too vague a statement to start choosing a test from. You need to more clearly express what the research question is/what you need to find out/what you're doing this for.

However, the data is highly skewed and t-test is obviously not a choice.

It seems like the responses for the two groups are strictly non-negative quantities.

What are the data measuring? What values are possible and impossible for the variable?

Why did you consider a t-test in the first place? Your initial sentence (you simply expressed a wish to compare two distributions) gives no hint that a t-test would be a suitable choice; it's specifically about comparing means under a set of conditions.

As an alternative, I wanted to use the MWW test for locational shift, but one of the assumptions is that the distributions of the groups should be similar, but in my case one group has a much longer tail than the other.

A location shift alternative usually doesn't make sense with a non-negative variable. I'd say more if I knew more about the variable, but you should probably consider whether a different sort of alternative makes sense (e.g. a scale-shift alternative often makes sense in such cases).

If you considered a t-test, you were apparently interested in comparing means; that's still possible with a scale-shift, given any of a large class of suitable models (or can be done nonparametrically).

If you don't insist on restricting yourself to a location-shift alternative, the WMW would be fine as long as it corresponds to your alternative of interest.

My last choice was the KS test used to compare arbitrary distributions.

Given your earlier statement that you sought a location shift alternative, that makes no sense -- you definitely would not be getting a pure location-shift alternative with this test (it rejects other things as well). If you want a "tends to be bigger" alternative, WMW would do that. If you want a scale shift alternative (which I expect will make a lot of sense once you explain what you're measuring), then you could easily go back to a comparison of means.

However, after running the test,

Then it seems you're done in any case, there's nothing more to be done now.

If you did a test already, you don't get to backpedal and choose a different test because the one you did was not powerful enough to reject. To choose to do another test in that circumstance would seem to be p hacking.

What troubles me is that I argued not to use MWW because the distributions differ,

Your problem was insisting on a location shift alternative (which is the only reason you'd need them to be the same shape under the alternative), an added requirement (above what the WMW needs) that you were nonetheless happy to abandon when moving to the KS test instead.

This inconsistency (not using WMW because it didn't appear to satisfy a condition you immediately abandoned anyway) seems strange to say the least.

Can somebody explain if my reasoning is valid, or is there something I forgot to consider?

When looking to do a test (assuming a test actually answers the question of interest):

  1. First figure out what sort of alternatives (a) make sense to consider given your variable, and (b) that would be of interest.

    (Why insist on a location shift alternative? Why then abandon it so readily?)

  2. Then choose a suitable distributional model (or choose to avoid any specific distributional model). On that basis, choose a suitable test.

  3. Don't use more than one test.

Additional comment: As a general rule, using the specific data set (e.g. via your boxplot) to inform your test choice may be problematic, as choosing the specific hypothesis on the basis of the exact same sample values you want to test it on impacts the properties of the test. If you can inform yourself about how the variable should behave (at least under H0) without having to do that, so much the better.

$\endgroup$
4
  • $\begingroup$ The data show time in minutes (e.g. 1.5 is 1 min and 30 sec) for completing some task, and I wanted to see if the time needed differs between the two groups of people in the sample. Does one group differ significantly from the other? I guess this also sounds vague by your standards. I know that KS is used to compare distributions, and judging from your comment, it shouldn't be considered as an alternative to testing means or medians (to which I agree). Since none of the tests I considered (and with which I'm familiar with) are not suitable, can you suggest some which would make sense? $\endgroup$
    – m_delx
    Jun 16, 2023 at 8:38
  • $\begingroup$ "you definitely would not be getting a location-shift alternative with this (KS) test." The two-sample KS-test tests whether distributions are equal against any alternative, including location shift. It may have worse power for that than WMW or t, but I don't agree with "definitely would not get". $\endgroup$ Jun 16, 2023 at 9:20
  • $\begingroup$ @ChristianHennig The trouble I see is that KS can reject for all kinds of reasons, so there is little reason to think a rejection indicates a location shift when the culprit just as easily could be unequal scale. $\endgroup$
    – Dave
    Jun 16, 2023 at 11:29
  • $\begingroup$ @Dave I think we all agree that KS isn't a good choice if it is of interest whether one distribution tends to produce larger values than the other. My comment was just technical; KS will reject in case there is location shift and the sample is big enough for having enough power (even though the power is in this case quite certainly weaker than WMW). $\endgroup$ Jun 16, 2023 at 11:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.