In articles such as "Research #1: Generative Models" one reads once more statements such as
These images are examples of what our visual world looks like and we refer to these as "samples from the true data distribution"
the images meant are the ImageNet dataset.
We also read statements such as
Mathematically, we think about a dataset of examples $x_1,\dots,x_n$ as samples from a true data distribution $p(x)$. In the example image below, the blue region shows the part of the image space that, with a high probability (over some threshold) contains real images, and black dots indicate our data points (each is one image in our dataset).
Our goal then is to find parameters $\theta$ that produce a distribution that closely matches the true data distribution (for example, by having a small KL divergence loss).
Whenever I read such language:
I wonder if statements such as the first one are not exaggerations in the sense that there is a huge number of 'real' images that could be in the dataset but are not in it. Or in other words that the ImageNet dataset is not representative of 'real image' even though it obviously samples it.
Being a real image does not seem to be a probabilistic matter to me. An image either came from the real world or did not. It does not matter that mechanical turk is used to make people guess if the machine generated images look real like in the article. This does not justify the 'probabilistic' interpretation of what the machine learning algorithm is doing. In other words, don't statements such as with a high probability (over some threshold) actually point to abuse or sweeping under the rug?
I wonder if statements such as the last one are not stretching it. We do not know the 'true data distribution', we only know a small number of samples of it. Using samples to measure loss is just a (huge) assumption about the relation of the samples to the distribution. That relation is swept under the rug.
I wonder if it is not possible to interpret what the machine learning algorithm does that does not rely on such terminology? Per example simply talking about compression? And that the generalisation produced is merely an assumption on the continuity between the samples (regularity)?
I know my questions are vague, but I would like to know if other people also have such thoughts, if they are well founded, if they are treated explicitly in some book or paper that I can't seem to find.