Clarification on the IID assumption in machine learning: who is sampled from where, and who is independent with who?

Question

So there are a couple of questions on IID assumption on this stackexchange,

On the importance of the i.i.d. assumption in statistical learning

Realistically, does the i.i.d. assumption hold for the vast majority of supervised learning tasks?

How can the IID assumption be checked in a given dataset?

How to generate data in order to fit the i.i.d. assumption in many machine learning applications?

What exactly is p(x,y) in the context of iid assumption in machine learning?

I just want to clarify, mathematically, what it means for people say that a data set $\{(x_i, y_i)\}_{i =1, \ldots, n}$ is i.i.d. (or sampled in an i.i.d. fashion)

So my question is simply, how does this translate mathematically?

My current working definition:

• (Identically distributed) Each sample $(x_i,y_i)$ is assumed to be sampled from a joint probability distribution $p(x_i,y_i)$ or in other words, each $(x_i, y_i)$ is the realization of the random variable $(X,Y)\sim p_{X,Y}(x_i,y_i)$

• (Independent) For $i \neq j$, the realization $(x_i,y_i)$ is generated independently from $(x_j, y_j)$

Why I am unsatisfied/unsure about my definitions:

1. For identically distributed, this question seems to say that each sample is drawn from the probability distribution over all the sample. That is, we assume that each data point $(x_i, y_i)$ is generated from a random variable $(X_i, Y_i)$, and that $(x_i,y_i)$ is not sampled from the distribution of $(X_i, Y_i) \sim p_{X,Y}(x_i,y_i)$ but from the distribution of all random variables $(X_1, \ldots, X_n, Y_1, \ldots, Y_n) \sim p(x_1, \ldots, x_n, y_1, \ldots, y_n)$ (subscript omitted). Which is different from the definition I have given above.

This is exactly what the notation $({\bf{X}}_i,y_i) \sim \mathbb{P}({\bf{X}},y), \forall i=1,...,N$ means in the linked question.

Which one is correct?

2. For independence: since $(x_i, y_i)$ and $(x_j,y_j)$ are just pairs of vectors, which are not random variables, hence we cannot speak about their independence. So to me, independence means that given $(x_i, y_i)$ and $(x_j,y_j)$ are generated by two (pairs of) random variables, $(X_i, Y_i)$ and $(X_j, Y_j)$, then the joint distribution $p_{X_i, X_j, Y_i, Y_j}(x_i, x_j, y_i, y_j)$ can be decomposed into $p_{X_i, Y_i}(x_i, y_i)p_{X_j, Y_j}(x_j, y_j)$.

   Is this correct?

Just want to be mathematically rigorous about things. Any reference will help me!!!

Answer 1

These are essentially the same definition

My current working definition:

• (Identically distributed) Each sample $(x_i,y_i)$ is assumed to be sampled from a joint probability distribution $p(x_i,y_i)$ or in other words, each $(x_i, y_i)$ is the realization of the random variable $(X,Y)\sim p_{X,Y}(x_i,y_i)$

• (Independent) For $i \neq j$, the realization $(x_i,y_i)$ is generated independently from $(x_j, y_j)$

The first part of your definition describes the marginal distribution of each data point. The second part describes the relationship between the datapoints; given this information, one can uniquely obtain the joint distribution of the entire dataset.

1. For identically distributed, this [question][1] seems to say that each sample is drawn from the probability distribution over all the sample. That is, we assume that each data point $(x_i, y_i)$ is generated from a random variable $(X_i, Y_i)$, and that $(x_i,y_i)$ is not sampled from the distribution of $(X_i, Y_i) \sim > p_{X,Y}(x_i,y_i)$ but from the distribution of all random variables $(X_1, \ldots, X_n, Y_1, \ldots, Y_n) \sim p(x_1, \ldots, x_n, y_1, > \ldots, y_n)$ (subscript omitted). Which is different from the definition I have given above.

This definition of i.i.d. is describing the joint distribution of the entire data set. Note that we can obtain the marginal distribution from the joint distribution just by integrating out the other terms, and that we can sample from the marginal distribution by sampling from the joint distribution then ignoring the other terms. Thus, both formulations are the same.

2. For independence: since $(x_i, y_i)$ and $(x_j,y_j)$ are just pairs of vectors, which are not random variables, hence we cannot speak about their independence. So to me, independence means that given $(x_i, y_i)$ and $(x_j,y_j)$ are generated by two (pairs of) random variables, $(X_i, Y_i)$ and $(X_j, Y_j)$, then the joint distribution $p_{X_i, X_j, Y_i, Y_j}(x_i, x_j, y_i, y_j)$ can be decomposed into $p_{X_i, Y_i}(x_i, y_i)p_{X_j, Y_j}(x_j, y_j)$. Is this correct?

Yes.