# Is the machine learning community abusing “true distribution"?

In articles such as "Research #1: Generative Models" one reads once more statements such as

These images are examples of what our visual world looks like and we refer to these as "samples from the true data distribution"

the images meant are the ImageNet dataset.

We also read statements such as

Mathematically, we think about a dataset of examples $x_1,\dots,x_n$ as samples from a true data distribution $p(x)$. In the example image below, the blue region shows the part of the image space that, with a high probability (over some threshold) contains real images, and black dots indicate our data points (each is one image in our dataset).

and

Our goal then is to find parameters $\theta$ that produce a distribution that closely matches the true data distribution (for example, by having a small KL divergence loss).

1. I wonder if statements such as the first one are not exaggerations in the sense that there is a huge number of 'real' images that could be in the dataset but are not in it. Or in other words that the ImageNet dataset is not representative of 'real image' even though it obviously samples it.

2. Being a real image does not seem to be a probabilistic matter to me. An image either came from the real world or did not. It does not matter that mechanical turk is used to make people guess if the machine generated images look real like in the article. This does not justify the 'probabilistic' interpretation of what the machine learning algorithm is doing. In other words, don't statements such as with a high probability (over some threshold) actually point to abuse or sweeping under the rug?

3. I wonder if statements such as the last one are not stretching it. We do not know the 'true data distribution', we only know a small number of samples of it. Using samples to measure loss is just a (huge) assumption about the relation of the samples to the distribution. That relation is swept under the rug.

4. I wonder if it is not possible to interpret what the machine learning algorithm does that does not rely on such terminology? Per example simply talking about compression? And that the generalisation produced is merely an assumption on the continuity between the samples (regularity)?

I know my questions are vague, but I would like to know if other people also have such thoughts, if they are well founded, if they are treated explicitly in some book or paper that I can't seem to find.

• I am not sure what you are asking, but let me comment on #1. It is not important that the samples represent our world or not. What's important is the process that generated those images. We are in general taking pictures of things that are relevant to people. There is a huge part of electromagnetic spectrum that we cannot even see. As long as that process is stable (changing very slowly over time), meaning that we continue taking pictures of similar things, the probability distribution is also stable. Here the distribution is a conditional distribution (conditioned on our taste). Jan 3 '17 at 11:54
• I know my questions are vague, so thanks for the comment. I think that the imagenet pictures are much more conditioned than to our taste. They seem to be conditioned to things like 'one main object per picture', 'portrait style' per example, and given that, I am not sure what 'true' here means. Is it then not the 'true distribution of pictures taken such that one mainly has a main object per picture taken in portrait style'? You see how that not being mentioned can be confusing to the novice reader .... Jan 3 '17 at 12:05
• Another point I made is that I don't see the probabilistic component and it seems to be forced. An image is either yes taken or not taken from the real world, there is no probability around that and no thresholds. Do you have a comment on that? Jan 3 '17 at 12:07
• the 'type' being what? 'real' and 'unreal' ? Jan 3 '17 at 12:14
• I am a statistician and I don't want to get into the issues of machine learning vocabulary. I think there is of course an overlap with probability and statistics but it is developing into a separate discipline. I think however that this is a thoughtful question that the OP raises. Just a short while ago today I was looking at a question in statistics where the argument was about "true models" and the mathematical properties that we attribute to assumptions placed on models for which formal results such as consistency and unbiasedness etc can be made from these idealizations. Jan 3 '17 at 15:28

1. I wonder if statements such as the first one are not exaggerations in the sense that there is a huge number of 'real' images that could be in the dataset but are not in it. Or in other words that the ImageNet dataset is not representative of 'real image' even though it obviously samples it.

And this is exactly what your quotes say, that those are samples from the distribution.

1. Being a real image does not seem to be a probabilistic matter to me. An image either came from the real world or did not. It does not matter that mechanical turk is used to make people guess if the machine generated images look real like in the article. This does not justify the 'probabilistic' interpretation of what the machine learning algorithm is doing. In other words, don't statements such as with a high probability (over some threshold) actually point to abuse or sweeping under the rug?

This depends on your definition of probability. It seems that in such cases as described above they implicitly adopt Bayesian understanding of probability, i.e. probability is a measure of how much believable something is. In such case you can talk about probabilities of non-repeatable events, for example what is the probability that the sun will rise tomorrow?. It will, or it will not, we have no data about such events happening in the past, so under a classical definition of probability this would not be a probabilistic problem, but it can be seen as probabilistic problem under Bayesian understanding of probability.

1. I wonder if statements such as the last one are not stretching it. We do not know the 'true data distribution', we only know a small number of samples of it. Using samples to measure loss is just a (huge) assumption about the relation of the samples to the distribution. That relation is swept under the rug.

And this is basically what statistics is about: we infer about properties of the population given only the limited sample from it, that we have. We learn about properties of the true distribution given only the empirical distribution obtained from our sample.

1. I wonder if it is not possible to interpret what the machine learning algorithm does that does not rely on such terminology? Per example simply talking about compression? And that the generalization produced is merely an assumption on the continuity between the samples (regularity)?

Yes, other models are possible, but it seems that the quotes you provided refer to probabilistic models. Again, statistics are grounded on probabilistic reasoning about data. Without probability it becomes much more problematic to quantify the uncertainty about the true distribution that rises from the fact that we are dealing with limited and imperfect data to infer about it.

This paragraph from a blog post seems to answer the question:

The key relationship between images and statistics is that we can interpret images as samples from a high-dimensional probability distribution. The probability distribution goes over the pixels of images. Imagine you’re taking a picture with your camera. This picture will have some finite number of pixels. When you take an image with your camera, you are sampling from this complex probability distribution. This distribution is what we’ll use to define what makes an image normal or not. With images, unlike with the normal distributions, we don’t know the true probability distribution and we can only collect samples.

Upon more thinking, what cements the idea for me is that one can in fact make any wanted pattern be a 'real picture', by drawing that pattern and with proper conditions, taking a picture of it. Hence, all combinations of pixels are 'real'. With this said, it is obvious that the probabilistic component is a necessity for a 'fake'/'real' classification.