How do test whether two multivariate distributions are sampled from the same underlying population?

Question

Say you are given two multivariate data sets, say an old one and a new one, and that they are supposed to have been generated by the same process(that you have no model for) but perhaps, somewhere along the line of collecting/creating the data, something went awry. You wouldn't want to use the new data as, say, a validation set for the old data or to add to the old data.

You can do a bunch of 1-d stats (per variable), e.g Wilcoxon rank sum, and try some multiple test correction but I'm not sure that's optimal (to capture the intricacies of multivariate data let alone multi-test issues). One way is to use a classifier and see if you can discriminate between the two datasets (given an optimal classifier that's optimal). That does seem to work but still a) perhpas there's a better way b) It's not really designed to tell you why it's different (if nothing else it will use the best predictors and possibly miss other good predictors that were subsumbed by the better ones)

Answer 1

http://131.95.113.139/courses/multivariate/mantel.pdf

Discusses two possible ways of doing just that if your datasets are the same size. The basic approach is to compute a distance metric between your two observed matrixes. Then to determine if that distance is significant, you use a permutation test.

If your datasets are not the same size then you can use the cross-match test although it does not appear to be very popular. Instead of the cross-match test you can try up or down sampling your data so they are the same size, then using one of the approaches mentioned in the first paper.

Answer 2

Look up Hotelling's T^2, or if yoy have really high-dim data, look at this: http://normaldeviate.wordpress.com/2012/07/14/modern-two-sample-tests/