4
$\begingroup$

I have two distributions $F$ and $G$ that I conjecture to be identical, mathematically. Essentially, before investing time in mathematically proving that $F$ and $G$ are equal, I'd like to sanity check it by doing a simulation experiment.

So, given that I can easily simulate $X_i \sim F$ and $Y_i \sim G$, I'd like some test that checks $F = G$. The $X_i$ and $Y_i$ live in $\mathbb R^p$, and I'm in the process of checking that the marginals are equal in distribution by simulating many data-sets and checking that the Kolmogov-Smirnov test statistics are approximately uniformly distributed, but I'd like some assurance that I'm not being deceived by just looking at the marginals.

In addition to the marginals, I'm checking the distribution of the principal components of the generated data/some nonlinear transformations of the data, and so far everything points to equality.

If someone could point me to a method - hopefully with an existing implementation in R, or otherwise something easy to implement - that would be fantastic.

EDIT: To be clear, I'm asking for a multivariate test. I know the marginals aren't sufficient, I'm only checking them because if the marginals don't match then I don't need to check the joint.

Improve this question
$\endgroup$
4
  • $\begingroup$ Any given set of marginals is compatible with any number of joint distributions. Think of it as the "indeterminacy principle" in the field of Statistics. Consider the simplest case of two dependent normal random variables, that are, moreover, jointly normally distributed. Their correlation coefficient does not appear in their marginals, neither does it affect the mean and variance of each. So for the same two marginal normals, even if you restrict attention to them being jointly normally distributed, their joint distribution can be any bivariate normal with correlation coefficient in $(-1,1)$. $\endgroup$
    – Alecos Papadopoulos
    Commented Jan 10, 2014 at 3:11
  • $\begingroup$ @AlecosPapadopoulos yes, I understand that. That's the point of the question :) I want a test that looks at the joint rather than some finite collection of marginals. In principle, I could get away with checking $t^T X = t^T Y$ in distribution, which is why I've also been checking other marginals like the principal components as additional sanity checks. $\endgroup$
    – guy
    Commented Jan 10, 2014 at 5:04
  • $\begingroup$ You might want to follow the literature cited in this recent JMLR paper. In particular, the Rosenbaum paper, and the Hall and Tajvidi paper look interesting (I have not read them). $\endgroup$
    – tchakravarty
    Commented Jan 19, 2014 at 8:23
  • $\begingroup$ duplicate of stats.stackexchange.com/questions/30687/… $\endgroup$
    – Stéphane Laurent
    Commented Jul 1, 2017 at 15:27

1 Answer 1

Reset to default
2
$\begingroup$

I'll list a couple things that have been useful in my own research:

First - have you considered the Hotelling's T-squared test?

http://en.wikipedia.org/wiki/Hotelling's_T-squared_distribution

(for some reason the apostrophe throws off the link given above, so you'll have to copy and paste into the address bar)

This is the multivariate generalization of the standard t-test between two groups. An R implementation of this test is given in the R package Hotelling.

However, If you don't want to make any distributional assumptions in your data you could also use a permutation test such as the Multi Response Permutation Procedure (MRPP) which is a distance based permutation procedure. There's an R implementation of this in the vegan package. Look for the mrpp function. I would recommend keeping the default at euclidean distance, but you can note that using squared euclidean distance would be equivalent to basing the permutation test on the Hotelling T-Squared statistic if you feel more comfortable with that.

Between the two things I mentioned - I would recommend using the MRPP test more strongly.

Improve this answer
$\endgroup$
2
  • $\begingroup$ (+1) but I'm not sure how applicable these are for me. The distributions $F$ and $G$ are both highly multimodal, and moreover are supported on a very nonlinear set. I can make no distributional assumptions at all, and I am looking for more than a difference of means so I'm not sure distance-based non-parametric tests are applicable either. $\endgroup$
    – guy
    Commented Jan 11, 2014 at 2:00
  • $\begingroup$ Just a note that MRPP based on Euclidean distance corresponds more towards differences in medians than means, but you're right - if you're looking for more than just differences in the location of the distribution, there may be better things to consider. I know there exist multivariate extensions of the Kolmogov-Smirnov test, but I'm unfamiliar with any R software to carry it out. $\endgroup$
    – Samuel Benidt
    Commented Jan 11, 2014 at 2:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.