In context to supervised learning, I have been told that the training dataset and testing data set must be obtained from same distribution whichever it is. That is, for a given supervised learning algorithm, if training data is obtained from say, a normal distribution, then test dataset must also be obtained from normal distribution.
Why is this restriction?
How will the learning performance change when this restriction, if it really exists, is uplifted?
Your training dataset is supposed to be an unbiased representation for your test dataset, otherwise you'd have a problem.
Your objective is to train a model that can generalize to a data set that you haven't seen, and that can only happen if what you use for training is approximately what your model will eventually encounter. No machine learning can compensate for bad and invalid inputs. You don't compare Apple and Orange.
If they come from different distributions, it is like trying your best to approximate an apple and then test your outcome with an orange.
In learning, the purpose is to find a "hypothesis" (say a function) which approximates a unique target function. This task is carried out based on a set of realisation of the target function. These realisations are usually divided into a train and a test set. At this point, hopefully, it is clear that both the train and test set are generated by that unique target function. Hence, they may not come from two different distributions.
