2
$\begingroup$

I want to compare two distributions, to see if they are significantly different. They represent task time completions (so they range from 1 to around 1000 seconds) in two different months. They are not normally distributed. I want to see if their central tendencies are significantly different (at a first glance the mode, mean and median between the two months seem very close, just 3-4 seconds difference), but also to see if their shapes are similar (again, at a first glance, they look similar). I am currently carrying this analysis with SPSS 20. I have the Mann-Whitney test for testing central tendencies and the Kolmogorov-Smirnov test for the shape of the distribution, (although I have read that the K-S test is an overall comparison test for the distributions).

Also, in the first month I have 300,000 observations and in the second month 122,000 observations. So, a lot of data ... but disproportionate. Is this an impediment to running these tests, the fact that the sample sizes are not equal? I ran both Mann-Whitney and K-S and they both seem to reject the null. How much should I trust the results given my sample sizes? Do you suggest any alternative tests? Thanks

$\endgroup$
  • $\begingroup$ NB: When you use the Mann-Whitney test to detect a difference in central tendency between populations you assume they have the same shape. $\endgroup$ – Scortchi - Reinstate Monica Jan 11 '13 at 12:54
  • $\begingroup$ Hi, Thanks for the note. So, is there a test (such as K-S test) that tests shape? Or should I look at basic statistics? Kurtosis, Skewness etc. And if so (and this should be probably a different question) how different should Kurtosis and Skewness (the values) be in order to say that the shape is different? $\endgroup$ – DL10x Jan 11 '13 at 12:58
  • $\begingroup$ You have so much data there that just about anything you test is likely to come out significant. But with those numbers, you could do a very good estimate of the two distributions and contemplate the differences. FYI, neither the K-S nor the Mann-Whitney require equal sample sizes. $\endgroup$ – Placidia Jan 11 '13 at 13:34
  • $\begingroup$ If I ever wanted to test shape differences in a non-parametric setting my first thought would be to standardize the data & then perform the K-S test (by simulation, as I'd have estimated the locations & spreads from the data). $\endgroup$ – Scortchi - Reinstate Monica Jan 11 '13 at 15:41
  • $\begingroup$ Thanks Placidia. I have tried to find a distribution to fit them using Q-Q plots, but I didn't manage yet (This is I think the only way I know to estimate a distribution :D my theoretical background kinda sucks). The tasks have some sort of "time-stop", the users know that they must complete the tasks after 500s but they can still go on after that. The data range from 0 to 1000 and approximately half of the distribution is S-shaped (with a proeminent peak, 75% of tasks are here) and the other half is L-shaped. It does not really fit any of the well-known distributions. Suggestions welcome $\endgroup$ – DL10x Jan 11 '13 at 16:12
2
$\begingroup$

With such large sample sizes both tests will have high power to detect minor differences. The 2 distributions could be almost identical with a small difference in shape location that is not of practical importance and the tests would reject (because they are different).

If all you really care about is a statistically significant difference then you can be happy with the results of the KS test (and others, even a t-test will be meaningful with non-normal data of those sample sizes due to the Central Limit Theorem).

If you care about practical or meaningful differences then things become subjective, but you can compare using various plots to help you decide if you think there are differences that are enough to care about.

Another possibility is doing a visual test as documented in

 Buja, A., Cook, D. Hofmann, H., Lawrence, M. Lee, E.-K., Swayne,
 D.F and Wickham, H. (2009) Statistical Inference for exploratory
 data analysis and model diagnostics Phil. Trans. R. Soc. A 2009
 367, 4361-4383 doi: 10.1098/rsta.2009.0120

The vis.test function in the TeachingDemos package for R helps implement the test, but it can be done by hand as well.

Basically you create a bunch of graphs and then see if you can tell which is which. For your question one possibility would be to create a histogram of the 122,000 observations from the one month, then take several samples of 122,000 from the 300,000 observations of the other month and create histograms of each of those samples. Then present someone (or several someones) with all the histograms in random order and see if they can pick out the one that represents the second month. If they consistently pick out the correct graph then that says there is something visually different and you can further explore how they differ. If they don't pick out the correct graph then that suggests that while there may be a statistally significant difference, it is not important enough to distinguish them visually.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.