Assessing the significance of differences in distributions I have two groups of data.  Each with a different distribution of multiple variables.  I'm trying to determine if these two groups' distributions are different in a statistically significant way.  I have the data in both raw form and binned up in easier to deal with discrete categories with frequency counts in each.  
What tests/procedures/methods should I use to determine whether or not these two groups are significantly different and how do I do that in SAS or R (or Orange)?
 A: You might be interested in applying relative distribution methods.  Call one group the reference group, and the other the comparison group.  In a way similar to constructing a probability-probability plot, you can construct a relative CDF/PDF, which is a ratio of the densities.  This relative density can be used for inference.  If the distributions are identical, you expect a uniform relative distribution.  There are tools, graphical and statistical, to explore and examine departures from uniformity.  
A good starting point to get a better sense is Applying Relative Distrbution Methods in R and the reldist package in R.  For details, you'll need to refer to the book, Relative Distribution Methods in the Social Sciences by Handcock and Morris.  There's also a paper by the authors covering the relevant techniques.
A: One measure of the difference between two distribution is the "maximum mean discrepancy" criteria, which basically measures the difference between the empirical means of the samples from the two distributions in a Reproducing Kernel Hilbert Space (RKHS). See this paper "A kernel method for the two sample problem".
A: I believe that this calls for a two-sample Kolmogorov–Smirnov test, or the like.  The two-sample Kolmogorov–Smirnov test is based on comparing differences in the empirical distribution functions (ECDF) of two samples, meaning it is sensitive to both location and shape of the the two samples.  It also generalizes out to a multivariate form.
This test is found in various forms in different packages in R, so if you are basically proficient, all you have to do is install one of them (e.g. fBasics), and run it on your sample data.
A: I'm going to ask the consultant's dumb question.  Why do you want to know if these distributions are different in a statistically significant way?  
Is it that the data that you are using are representative samples from populations or processes, and you want to assess the evidence that those populations or processes differ?  If so, then a statistical test is right for you.  But this seems like a strange question to me.
Or, are you interested in whether you really need to behave as though those populations or processes are different, regardless of the truth?  Then you will be better off determining a loss function, ideally one that returns units that are meaningful to you, and predicting the expected loss when you (a) treat the populations as different, and (b) treat them as the same.  Or you can choose some quantile of the loss distribution if you want to adopt a more or less conservative position.
A: I don't know how to use SAS/R/Orange, but it sounds like the kind of test you need is a chi-square test. 
