In statistics and machine learning, a common starting point is to assume some unknown distribution $\mathbb{P}$ on the cartesian product $\mathcal{X} \times \mathcal{Y}$ of input space and output space. The training data $\{x_i, y_i\}_{i = 1, \ldots , n}$ is then assumed to be i.i.d. samples from $\mathbb{P}$.

What does it mean to not have this assumption? Specifically, does it make sense to have an assumption which says that there is no distribution on $\mathcal{X} \times \mathcal{Y}$ that generated the training sample? Is learning possible in this regime?

Of course, we can always assume that the training data was sampled from the empirical distribution $\frac{1}{n}\sum_{i=1}^n \delta_{(x_i, y_i)}$, and so there always exists a distribution on $\mathcal{X} \times \mathcal{Y}$ that could have generated the sample, but this seems too trivial to be useful or interesting.

The original assumption is studied under the label of "distribution free setting", and perhaps what I am asking is "Is there literally a distribution free setting studied in statistics or ML?"

  • 4
    $\begingroup$ “Distribution-free” means that the distribution is unspecified (could be normal, could be exponential, whatever), but there is some distribution. Did you come across a source suggesting otherwise? $\endgroup$
    – Dave
    Commented Jul 22, 2021 at 1:20
  • 1
    $\begingroup$ Yes, I understand that. No, I didn't come across any source suggesting otherwise. I am just curious to know if there exists other "models" of statistics. $\endgroup$
    – Aditya
    Commented Jul 22, 2021 at 1:23
  • 4
    $\begingroup$ First have a look at stats.stackexchange.com/questions/445453/… to understand iid assumption better. But there are certainly other models for dependent data: time series, spatial data, repeated measurements, ... $\endgroup$ Commented Jul 22, 2021 at 2:47
  • 4
    $\begingroup$ If you want to use data to predict new values, you must posit some relation connecting the old to the new. Regardless, this reads like a misunderstanding of the (awful and misleading) term "distribution-free," as Dave has pointed out. $\endgroup$
    – whuber
    Commented Jul 23, 2021 at 21:08
  • $\begingroup$ @Ben & @ kjetil b halvorsen, the link is indeed very useful, thanks. I understand the notion of exchangeability to relax the strong assumption of independence. Are there other notions of probabilistic symmetries which have been studied specifically in the context of statistics and machine learning? $\endgroup$
    – Aditya
    Commented Jul 24, 2021 at 20:42

1 Answer 1


Your question is only a slight variation (and arguably a duplicate) of a question here. At its most general, your problem will involve a joint distribution for the vector $(x_1,...,x_n,y_1,...,y_n)$, but if the pairs of values in this vector are not IID then this will not reduce to a common distribution for each $(x_i,y_i)$, and so there will be no unambiguous distribution of interest on $\mathcal{X} \times \mathcal{Y}$.

If you assume that the IID assumption holds, so that you have a common distribution for each pair $(x_i,y_i)$, you then have to decide if you want to impose any structure on the distribution (e.g., an assumption that it is of a particular parametric family of distributions). If no further structure is imposed, such that this is just an arbitrary bivariate distribution, some sources will call this a "distribution-free" analysis. (I have always hated this terminology; what they really mean is that they are not using any parametric family of distributions, but they are still using a distribution in a broader space.)

Even in the case where you don't use the IID assumption, statistical modelling will still assume that there is some joint distribution over the vector $(x_1,...,x_n,y_1,...,y_n)$ containing all the observable quantities in the analysis. This comes from the underlying assumption that these we are working within some probability space, and the observable variables are numeric labels on the sample space (i.e., mappings from the sample space to real numbers). In this case there is a single $2n$-variate distribution instead of a common bivariate distribution. You can't really do much in this case, since you will only have one observation from this distribution, so inference of its structure is essentially impossible.

Now, it is of course possible to go further than this, and jettison the entire assumption that we are working within a probability space. If we do this then there are no probability distributions in the analysis --- this would be a distribution-free model in the literal sense. (Note that certain "distribution-free" models are not distribution-free in the literal sense; they just eschew parametric families of distributions.) However, this removes the entire apparatus of probability as a tool of analysis, and so any consequent statistical or machine learning work would then be uninformed by probability theory. This is possible of course, but it is a bit like performing surgery without use of the underlying biology of the human body; it is messy, and can lead to disaster.

  • $\begingroup$ This discussion is exactly what I was looking for. Can you expound or suggest some resources on your point about working outside a probability space? Is there another "philosophy" of statistics which does not rely on probability theory? $\endgroup$
    – Aditya
    Commented Jul 24, 2021 at 20:35
  • $\begingroup$ There are a few paradigms floating around for "statistics without probability", but these are not widely accepted, and they raise obvious interpretive questions that are difficult for their proponents to answer. $\endgroup$
    – Ben
    Commented Jul 26, 2021 at 22:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.