23
$\begingroup$

I have two groups of data. Each with a different distribution of multiple variables. I'm trying to determine if these two groups' distributions are different in a statistically significant way. I have the data in both raw form and binned up in easier to deal with discrete categories with frequency counts in each.

What tests/procedures/methods should I use to determine whether or not these two groups are significantly different and how do I do that in SAS or R (or Orange)?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Are you interested in whether the distributions are of a different form (e.g., normal, poisson, etc.) or whether parameters are different (e.g., mean or sd of a normal distribution) or both? $\endgroup$
    – Jeromy Anglim
    Commented Sep 8, 2010 at 3:16
  • $\begingroup$ A related question: stats.stackexchange.com/questions/9311/… $\endgroup$
    – GaBorgulya
    Commented Apr 10, 2011 at 15:09

5 Answers 5

Reset to default
19
$\begingroup$

I believe that this calls for a two-sample Kolmogorov–Smirnov test, or the like. The two-sample Kolmogorov–Smirnov test is based on comparing differences in the empirical distribution functions (ECDF) of two samples, meaning it is sensitive to both location and shape of the the two samples. It also generalizes out to a multivariate form.

This test is found in various forms in different packages in R, so if you are basically proficient, all you have to do is install one of them (e.g. fBasics), and run it on your sample data.

Improve this answer
$\endgroup$
2
  • 6
    $\begingroup$ For R ks.test in the default "stats" package can conduct the KS test without installing additional packages. $\endgroup$
    – russellpierce
    Commented Jul 21, 2010 at 0:23
  • $\begingroup$ In SAS, KS test is available in proc npar1way. In R, in addition to ks.test(), there is the nortest package which provides several other adjustment tests. $\endgroup$
    – chl
    Commented Sep 8, 2010 at 11:15
11
$\begingroup$

I'm going to ask the consultant's dumb question. Why do you want to know if these distributions are different in a statistically significant way?

Is it that the data that you are using are representative samples from populations or processes, and you want to assess the evidence that those populations or processes differ? If so, then a statistical test is right for you. But this seems like a strange question to me.

Or, are you interested in whether you really need to behave as though those populations or processes are different, regardless of the truth? Then you will be better off determining a loss function, ideally one that returns units that are meaningful to you, and predicting the expected loss when you (a) treat the populations as different, and (b) treat them as the same. Or you can choose some quantile of the loss distribution if you want to adopt a more or less conservative position.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Your tone is a little snarky and condescending... but you're right, i think what i was really after was whether or not i can reasonably assume the two distributions are the same. $\endgroup$
    – Jay Stevens
    Commented Jun 22, 2011 at 12:41
  • 5
    $\begingroup$ Sorry that you don't like my tone. If you want to know whether you can reasonably assume that the two distributions are the same, then the KS will mislead you, because it tests the null hypothesis that the two distributions are the same. $\endgroup$
    – Andrew Robinson
    Commented Aug 16, 2011 at 11:43
8
$\begingroup$

You might be interested in applying relative distribution methods. Call one group the reference group, and the other the comparison group. In a way similar to constructing a probability-probability plot, you can construct a relative CDF/PDF, which is a ratio of the densities. This relative density can be used for inference. If the distributions are identical, you expect a uniform relative distribution. There are tools, graphical and statistical, to explore and examine departures from uniformity.

A good starting point to get a better sense is Applying Relative Distrbution Methods in R and the reldist package in R. For details, you'll need to refer to the book, Relative Distribution Methods in the Social Sciences by Handcock and Morris. There's also a paper by the authors covering the relevant techniques.

Improve this answer
$\endgroup$
3
$\begingroup$

One measure of the difference between two distribution is the "maximum mean discrepancy" criteria, which basically measures the difference between the empirical means of the samples from the two distributions in a Reproducing Kernel Hilbert Space (RKHS). See this paper "A kernel method for the two sample problem".

Improve this answer
$\endgroup$
1
  • $\begingroup$ This method is most robust in my opinion but not well known as it works equally well if you have finite sample for your distribution (and thus your sample distributions are not entirely continuous). It also works with multinomial distributions which for a KS test is still active research as far as I'm aware $\endgroup$
    – www3
    Commented Jun 30, 2017 at 19:12
-1
$\begingroup$

I don't know how to use SAS/R/Orange, but it sounds like the kind of test you need is a chi-square test.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I thought Chi-Sq was primarily for categorical data (contingency tables) vs. continuous? $\endgroup$
    – Jay Stevens
    Commented Jul 21, 2010 at 14:17
  • 1
    $\begingroup$ Hmmm I actually like the KS test answer better than mine ! $\endgroup$
    – Suresh Venkatasubramanian
    Commented Jul 22, 2010 at 0:07
  • 1
    $\begingroup$ No, this is not correct. $\endgroup$
    – SmallChess
    Commented Aug 27, 2016 at 8:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.