It is said that traditional machine learning often makes an assumption that both the training and test data are drawn from the same distribution. But it isn't clear what this means in a practical sense. For instance, what would this mean in the case of image data, such as people's faces for facial recognition? In that example, why would one set of images of people's faces be "drawn from a different distribution" than another set of images of people's faces? I am looking for an explanation that connects this mathematical language of "drawn from the same distribution" to practical, real-world examples.
$\begingroup$
$\endgroup$
3
asked Sep 17, 2020 at 3:56
-
$\begingroup$ which ML textbook says there is an assumption that both the training and test data are drawn from the same distribution? And for which ML algorithms? $\endgroup$– develaristCommented Sep 17, 2020 at 10:27
-
$\begingroup$ @develarist Chapter 1.1 of Transfer Learning by Sinno Jialin Pan, Qiang Yang, Wenyuan Dai, Zhang Yu. $\endgroup$– The PointerCommented Sep 17, 2020 at 10:50
-
$\begingroup$ I'll comment this as an answer $\endgroup$– develaristCommented Sep 17, 2020 at 14:07
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
It means that the test data look like the training data. For example, if your facial recognition system was developed in China, it might work well in China, but not if you try to use it in a country where people look different.
The "drawn" part means that the data is sampled at random from some population of interest. For example, the population could be everybody in China, and you assume that everyone has an equal chance of being in your training data. This might not be true, but it could be close enough to have a reasonable useful model.
The "distribution" part is what I called the "population of interest" in the previous paragraph.
This assumption is crucial when you try to evaluate the model. Usually this is done by splitting off a part of the data, chosen randomly, and using it to test the model. But if the model is evaluated in this way, then in the future it needs to be applied to data that looks like the data which was originally put into it, or else any claims about the model's performance will be misleading.
For example, it could be that your facial recognition system is 90% accurate when applied to Chinese faces, but only 50% accurate when applied to Russian faces. But you wouldn't have known that in advance, because your test data consisted only of Chinese faces.
In the case of facial recognition, it's fairly obvious what will go wrong if you apply it to unseen data. But this problem also happens for other models, often because future data doesn't look like past data. For example, banks have to re-do their credit risk models every year. This is a huge project which provides employment for a lot of people. But (as an auditor once said to me) if the models work, why do they have to be re-done every year? And if they don't work, why make the effort to build them at all?
answered Sep 17, 2020 at 4:53
$\begingroup$
$\endgroup$
It is not an assumption.
The careless statement Pan makes is not an assumption since no other data scientist seems to generally assert it given how large a field ML is.
What he is trying to say, and should be corrected, is that ML performs best when the test data is similar to the training data. This makes much more practical sense, and would be corroborated by others. From a probabilistic perspective of ML, this corrected statement translates to the test data having the same distribution as the training data. Stipulating the conditions under which a family of procedures works best under is by no way an assumption
answered Sep 17, 2020 at 14:10