Difference between training and test data distribution

Question

The basic assumption in machine learning is training and test data follows same distribution. But in reality this is highly unlikely. Covariate shift address this issue in which training and test distributions are different. Can someone clear the following doubts regarding this ?

1. How to check whether two distribution are different statistically ?
2. Can kernel density estimate (KDE) approach be used to estimate the probability distribution to tell the difference ?
3. Lets say I have 100 images of a specific category. Number of test images is 50. I'm changing the number of training images from 5 to 50 in steps of 5. Can I say the probability distributions are different when using 5 training images and 50 testing images after estimating them by KDE?

Answer 1

Ordinarily, you would obtain your training data as a simple random sample of your total dataset. This allows you to take advantage of all the known properties of random samples, including the fact that the training and test data then have the same underlying distributions. Indeed, the main purpose of this split is to use one set of data to "train" your model (i.e., fit the model) and the other set of data to set hypotheses of interest in that model. If you do not randomly sample your training data then you get all sorts of problems arising from the fact that there may be systematic differences between the two parts of your data.

Answer 2

I think you're confusing the underlying distribution from which both training and test distributions are drawn, with the distributions of the specific train and test draws.

Unless the underlying distribution is eg time-sensitive, changed during the time between eg drawing the training and the testing samples, the underlying distribution is identical each time.

The goal in learning a machine learning model is typically not to learn the training distribution, but to learn the latent underlying distribution, of which the training distribution is only a sample. Of course, you cannot actually see the underlying distribution, but eg, if you only really cared about learning the training samples, you could simply memorize the training samples in a lookup table, end of story. In reality, you are using the training sample as a proxy into the underlying distribution. "Generalization" is a somewhat synonym for "try to learn the underlying distribution, rather than just overfitting to the training samples".

To estimate how well the training data, and your fitted model, matches the underlying distribution, one approach is to draw one training set, one test set. Train on the training set, test on the test set. In reality, since you're most likely fitting a bunch of hyperparameters, you'll overfit these against the test set, think you're getting some super awesome mega accuracy, then fail horribly when you put the model into production.

A better approach is to use cross-fold validation:

• draw a bunch of training data
• split it randomly into 80% training data, 20% valdiation/dev data
  • run training/test on this, note down the accuracy etc
• redo the split, eg using a different random seed
  • re-run train/evaluate
• redo eg 5, 10, 20 times, depending on how much variance you are seeing
• this will give you a fairly realistic insight into how well your training sets and model are fitting the underlying distribution
• it's pretty general. You can use this approach for any i.i.d datasets