# Comparing huge data sets that are non-normal and different lengths in R

I'm comparing two data sets. Each set is extremely large, about 25,000 quantitative pieces of data in length. I want to find if these sets are significantly different, but the problem is that they are not normally distributed and are different lengths. I've tried a variety of tests in R: Mood's median test, Wilcoxan test, Kruskal-Wallis, but these seem to require either datasets of the same length or a normal distribution. I have not been able to find a test that works for the data I want to compare. Do you know of any statistical tests that can be used to compare my non-normally distributed, different-length data sets?

• (Be aware that asking for R code is off topic here.) What is it that you want to compare about them, location of their means, amount of variability, the shape of their distributions, something else? Commented Jun 23, 2016 at 17:08
• The phrase "quantitative piece of data" suggests you have more than numbers--but what do you have?
– whuber
Commented Jun 23, 2016 at 18:42

You may be looking for the two-sample Kolmogorov-Smirnov test, which assesses a measure of distance between the two samples' cumulative distribution functions. As such, it can be used for samples of different size. In R, look at ?ks.test.

However, of course with datasets this large, even small deviations in the CDF will be detected as statistically significant. Whether these are clinically significant cannot be assessed by statistical tests - look at quantiles, density plots, histograms and so forth for this.

Plus, if you have a specific question you are most interested in, like whether the means differ, or the variances (assuming equal means or not), of course more specialized tests are likely available, or you might be able to perform a nonparametric test, e.g., a permutation test.

Usually with large size of data, you want to explore and cleaning first. Try to do a summary in R and plot the histogram. Try to exam the range and outliers. It is almost impossible it will fit certain distribution without data clearing.

After you clean the data, you should try to see which distribution fits better and think about possible hypothesis testing.

PS, different length is not a problem at all. You are getting different samples, most hypothesis testing methods would not require the exactly same size on samples.

Also please note that, with large size, hypothesis testing will almost certain to tell you two samples are different, please see details in this post

You can trying a toy example to give you a feel about the importance of cleaning data. I am running 2 tests on if data is normally distributed. Adding 1 outlier will completely change the story.