Assessing the significance of differences in distributions

I have two groups of data. Each with a different distribution of multiple variables. I'm trying to determine if these two groups' distributions are different in a statistically significant way. I have the data in both raw form and binned up in easier to deal with discrete categories with frequency counts in each.

What tests/procedures/methods should I use to determine whether or not these two groups are significantly different and how do I do that in SAS or R (or Orange)?

• Are you interested in whether the distributions are of a different form (e.g., normal, poisson, etc.) or whether parameters are different (e.g., mean or sd of a normal distribution) or both? – Jeromy Anglim Sep 8 '10 at 3:16
• A related question: stats.stackexchange.com/questions/9311/… – GaBorgulya Apr 10 '11 at 15:09

I believe that this calls for a two-sample Kolmogorov–Smirnov test, or the like. The two-sample Kolmogorov–Smirnov test is based on comparing differences in the empirical distribution functions (ECDF) of two samples, meaning it is sensitive to both location and shape of the the two samples. It also generalizes out to a multivariate form.

This test is found in various forms in different packages in R, so if you are basically proficient, all you have to do is install one of them (e.g. fBasics), and run it on your sample data.

• For R ks.test in the default "stats" package can conduct the KS test without installing additional packages. – russellpierce Jul 21 '10 at 0:23
• In SAS, KS test is available in proc npar1way. In R, in addition to ks.test(), there is the nortest package which provides several other adjustment tests. – chl Sep 8 '10 at 11:15

I'm going to ask the consultant's dumb question. Why do you want to know if these distributions are different in a statistically significant way?

Is it that the data that you are using are representative samples from populations or processes, and you want to assess the evidence that those populations or processes differ? If so, then a statistical test is right for you. But this seems like a strange question to me.

Or, are you interested in whether you really need to behave as though those populations or processes are different, regardless of the truth? Then you will be better off determining a loss function, ideally one that returns units that are meaningful to you, and predicting the expected loss when you (a) treat the populations as different, and (b) treat them as the same. Or you can choose some quantile of the loss distribution if you want to adopt a more or less conservative position.

• Your tone is a little snarky and condescending... but you're right, i think what i was really after was whether or not i can reasonably assume the two distributions are the same. – Jay Stevens Jun 22 '11 at 12:41
• Sorry that you don't like my tone. If you want to know whether you can reasonably assume that the two distributions are the same, then the KS will mislead you, because it tests the null hypothesis that the two distributions are the same. – Andrew Robinson Aug 16 '11 at 11:43

You might be interested in applying relative distribution methods. Call one group the reference group, and the other the comparison group. In a way similar to constructing a probability-probability plot, you can construct a relative CDF/PDF, which is a ratio of the densities. This relative density can be used for inference. If the distributions are identical, you expect a uniform relative distribution. There are tools, graphical and statistical, to explore and examine departures from uniformity.

A good starting point to get a better sense is Applying Relative Distrbution Methods in R and the reldist package in R. For details, you'll need to refer to the book, Relative Distribution Methods in the Social Sciences by Handcock and Morris. There's also a paper by the authors covering the relevant techniques.

One measure of the difference between two distribution is the "maximum mean discrepancy" criteria, which basically measures the difference between the empirical means of the samples from the two distributions in a Reproducing Kernel Hilbert Space (RKHS). See this paper "A kernel method for the two sample problem".

• This method is most robust in my opinion but not well known as it works equally well if you have finite sample for your distribution (and thus your sample distributions are not entirely continuous). It also works with multinomial distributions which for a KS test is still active research as far as I'm aware – www3 Jun 30 '17 at 19:12

I don't know how to use SAS/R/Orange, but it sounds like the kind of test you need is a chi-square test.

• I thought Chi-Sq was primarily for categorical data (contingency tables) vs. continuous? – Jay Stevens Jul 21 '10 at 14:17
• Hmmm I actually like the KS test answer better than mine ! – Suresh Venkatasubramanian Jul 22 '10 at 0:07
• No, this is not correct. – SmallChess Aug 27 '16 at 8:59