Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have two groups of data. Each with a different distribution of multiple variables. I'm trying to determine if these two groups' distributions are different in a statistically significant way. I have the data in both raw form and binned up in easier to deal with discrete categories with frequency counts in each.

What tests/procedures/methods should I use to determine whether or not these two groups are significantly different and how do I do that in SAS or R (or Orange)?

share|improve this question
Are you interested in whether the distributions are of a different form (e.g., normal, poisson, etc.) or whether parameters are different (e.g., mean or sd of a normal distribution) or both? – Jeromy Anglim Sep 8 '10 at 3:16
A related question:… – GaBorgulya Apr 10 '11 at 15:09
up vote 8 down vote accepted

I believe that this calls for a two-sample Kolmogorov–Smirnov test, or the like. The two-sample Kolmogorov–Smirnov test is based on comparing differences in the empirical distribution functions (ECDF) of two samples, meaning it is sensitive to both location and shape of the the two samples. It also generalizes out to a multivariate form.

This test is found in various forms in different packages in R, so if you are basically proficient, all you have to do is install one of them (e.g. fBasics), and run it on your sample data.

share|improve this answer
For R ks.test in the default "stats" package can conduct the KS test without installing additional packages. – rpierce Jul 21 '10 at 0:23
Thanks for the information, drknexus. – John L. Taylor Jul 21 '10 at 16:53
In SAS, KS test is available in proc npar1way. In R, in addition to ks.test(), there is the nortest package which provides several other adjustment tests. – chl Sep 8 '10 at 11:15

I'm going to ask the consultant's dumb question. Why do you want to know if these distributions are different in a statistically significant way?

Is it that the data that you are using are representative samples from populations or processes, and you want to assess the evidence that those populations or processes differ? If so, then a statistical test is right for you. But this seems like a strange question to me.

Or, are you interested in whether you really need to behave as though those populations or processes are different, regardless of the truth? Then you will be better off determining a loss function, ideally one that returns units that are meaningful to you, and predicting the expected loss when you (a) treat the populations as different, and (b) treat them as the same. Or you can choose some quantile of the loss distribution if you want to adopt a more or less conservative position.

share|improve this answer
Your tone is a little snarky and condescending... but you're right, i think what i was really after was whether or not i can reasonably assume the two distributions are the same. – Jay Stevens Jun 22 '11 at 12:41
Sorry that you don't like my tone. If you want to know whether you can reasonably assume that the two distributions are the same, then the KS will mislead you, because it tests the null hypothesis that the two distributions are the same. – Andrew Robinson Aug 16 '11 at 11:43

You might be interested in applying relative distribution methods. Call one group the reference group, and the other the comparison group. In a way similar to constructing a probability-probability plot, you can construct a relative CDF/PDF, which is a ratio of the densities. This relative density can be used for inference. If the distributions are identical, you expect a uniform relative distribution. There are tools, graphical and statistical, to explore and examine departures from uniformity.

A good starting point to get a better sense is Applying Relative Distrbution Methods in R and the reldist package in R. For details, you'll need to refer to the book, Relative Distribution Methods in the Social Sciences by Handcock and Morris. There's also a paper by the authors covering the relevant techniques.

share|improve this answer

I don't know how to use SAS/R/Orange, but it sounds like the kind of test you need is a chi-square test.

share|improve this answer
I thought Chi-Sq was primarily for categorical data (contingency tables) vs. continuous? – Jay Stevens Jul 21 '10 at 14:17
Hmmm I actually like the KS test answer better than mine ! – Suresh Venkatasubramanian Jul 22 '10 at 0:07

One measure of the difference between two distribution is the "maximum mean discrepancy" criteria, which basically measures the difference between the empirical means of the samples from the two distributions in a Reproducing Kernel Hilbert Space (RKHS). See this paper "A kernel method for the two sample problem".

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.