# Meaning of NOT having a distribution on data [duplicate]

In statistics and machine learning, a common starting point is to assume some unknown distribution $$\mathbb{P}$$ on the cartesian product $$\mathcal{X} \times \mathcal{Y}$$ of input space and output space. The training data $$\{x_i, y_i\}_{i = 1, \ldots , n}$$ is then assumed to be i.i.d. samples from $$\mathbb{P}$$.

What does it mean to not have this assumption? Specifically, does it make sense to have an assumption which says that there is no distribution on $$\mathcal{X} \times \mathcal{Y}$$ that generated the training sample? Is learning possible in this regime?

Of course, we can always assume that the training data was sampled from the empirical distribution $$\frac{1}{n}\sum_{i=1}^n \delta_{(x_i, y_i)}$$, and so there always exists a distribution on $$\mathcal{X} \times \mathcal{Y}$$ that could have generated the sample, but this seems too trivial to be useful or interesting.

The original assumption is studied under the label of "distribution free setting", and perhaps what I am asking is "Is there literally a distribution free setting studied in statistics or ML?"

• “Distribution-free” means that the distribution is unspecified (could be normal, could be exponential, whatever), but there is some distribution. Did you come across a source suggesting otherwise?
– Dave
Commented Jul 22, 2021 at 1:20
• Yes, I understand that. No, I didn't come across any source suggesting otherwise. I am just curious to know if there exists other "models" of statistics. Commented Jul 22, 2021 at 1:23
• First have a look at stats.stackexchange.com/questions/445453/… to understand iid assumption better. But there are certainly other models for dependent data: time series, spatial data, repeated measurements, ... Commented Jul 22, 2021 at 2:47
• If you want to use data to predict new values, you must posit some relation connecting the old to the new. Regardless, this reads like a misunderstanding of the (awful and misleading) term "distribution-free," as Dave has pointed out.
– whuber
Commented Jul 23, 2021 at 21:08
• @Ben & @ kjetil b halvorsen, the link is indeed very useful, thanks. I understand the notion of exchangeability to relax the strong assumption of independence. Are there other notions of probabilistic symmetries which have been studied specifically in the context of statistics and machine learning? Commented Jul 24, 2021 at 20:42

Your question is only a slight variation (and arguably a duplicate) of a question here. At its most general, your problem will involve a joint distribution for the vector $$(x_1,...,x_n,y_1,...,y_n)$$, but if the pairs of values in this vector are not IID then this will not reduce to a common distribution for each $$(x_i,y_i)$$, and so there will be no unambiguous distribution of interest on $$\mathcal{X} \times \mathcal{Y}$$.
If you assume that the IID assumption holds, so that you have a common distribution for each pair $$(x_i,y_i)$$, you then have to decide if you want to impose any structure on the distribution (e.g., an assumption that it is of a particular parametric family of distributions). If no further structure is imposed, such that this is just an arbitrary bivariate distribution, some sources will call this a "distribution-free" analysis. (I have always hated this terminology; what they really mean is that they are not using any parametric family of distributions, but they are still using a distribution in a broader space.)
Even in the case where you don't use the IID assumption, statistical modelling will still assume that there is some joint distribution over the vector $$(x_1,...,x_n,y_1,...,y_n)$$ containing all the observable quantities in the analysis. This comes from the underlying assumption that these we are working within some probability space, and the observable variables are numeric labels on the sample space (i.e., mappings from the sample space to real numbers). In this case there is a single $$2n$$-variate distribution instead of a common bivariate distribution. You can't really do much in this case, since you will only have one observation from this distribution, so inference of its structure is essentially impossible.