When we fit a model to a training dataset, our basic assumption is that the test dataset will also come from the same (or similar) distribution. How to check if the distribution of the training and test dataset is same (similar)?

More specifically:

Let $(\textbf{x}, y) \sim \mathcal{D}$ where $\textbf{x} \in \mathbb{R^{5}}$ are independent variables and $y$ is the dependent variable and $\mathcal{D}$ is some unknown distribution.

Case I Regression: $y \in \mathbb{R}$

Case II Classification: $y \in \{0, 1\}$

Question: For both these specific cases, how do we test if both training and test dataset is drawn from $\mathcal{D}$? Alternatively, how do we test if the distribution of test and training dataset is not statistically different?

Related Question: Does change in data distribution affect non-parametric models such as "Nearest Neighbor" as much as it would affect the parametric models? My intuition is that it should not.

PS 1: I understand that we can't test if two datasets are from the same distribution, but we can test if they are from two statistically different distribution.

PS 2: Since this seems a broad question, even pointers to some good articles should be fine. The ones that I found online dealt with the case of $x \in \mathbb{R}$ instead of $\textbf{x} \in \mathbb{R}^{5}$ and did not differentiate regression from classification. Moreover it assumed a known distribution.


Case I and Case II aren't so different as you might think: just consider the joint distribution of $(\mathbf x, y)$, which you've helpfully already notated as $\mathcal D$. Then any two-sample testing procedure would work; I'm partial to the maximum mean discrepancy test described by

Gretton, Borgwardt, Rasch, Schölkopf, and Smola. A Kernel Two-Sample Test. JMLR 13(Mar):723−773, 2012.

It's particularly good if you (warning: shameless self-promotion) optimize the kernel with the technique proposed by the following paper:

Sutherland, Tung, Strathmann, De, Ramdas, Smola, and Gretton. Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy. ICLR 2017. (code)

There are also methods that propose to automatically adapt for possible changes between the training and the test distribution, perhaps with a few labeled examples in the test distribution. This field in general is known as transfer learning, and there's a fair amount of recent work in it; often it attempts to make the two distributions look similar to a two-sample test. A reasonable starting point is this 2016 PhD thesis (by a former labmate of mine), though there is also surely a lot of very recent deep learning work that isn't discussed there (and that I'm not very familiar with).

For your related question, I don't think it's the case that nonparametric models are in general more robust to distribution shift than parametric methods. You might be able to come up with examples where they are, but it's going to be highly dependent on the type of shift you assume; I'm sure you can also come up with examples where the nonparametric methods break faster than the parametric ones.

  • $\begingroup$ Dougal: Thank you. This is exactly what I was looking for. Glad to see that you have worked with Alex Smola and Aaditya Ramdas in the past!! I am sure that was a fun :) $\endgroup$ Feb 8 '18 at 22:00
  • $\begingroup$ I see that you work with Active Learning as well. I also have a related question to that. Let's say you learn a text classifier and then as you see new test data you measure the performance and also identify the cases where you did wrong. How can we retrain the classifier from labels that were incorrectly identified in the test dataset? Both for multi-class and binary class cases (if that makes any difference in the learning algorithm). $\endgroup$ Feb 8 '18 at 22:06
  • 1
    $\begingroup$ Let me post this question separately and provide a link here $\endgroup$ Feb 8 '18 at 22:06
  • $\begingroup$ @TheWanderer I think that's just online learning, no active necessarily required, assuming you always get the test labels afterwards. Active learning would be if you had to decide when to get those labels or not, and then it's a fairly standard problem setting there too. $\endgroup$
    – Danica
    Feb 8 '18 at 22:07
  • $\begingroup$ stats.stackexchange.com/questions/327635/… $\endgroup$ Feb 8 '18 at 22:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.