So there are a couple of questions on IID assumption on this stackexchange,
On the importance of the i.i.d. assumption in statistical learning
Realistically, does the i.i.d. assumption hold for the vast majority of supervised learning tasks?
How can the IID assumption be checked in a given dataset?
How to generate data in order to fit the i.i.d. assumption in many machine learning applications?
What exactly is p(x,y) in the context of iid assumption in machine learning?
I just want to clarify, mathematically, what it means for people say that a data set $\{(x_i, y_i)\}_{i =1, \ldots, n}$ is i.i.d. (or sampled in an i.i.d. fashion)
So my question is simply, how does this translate mathematically?
My current working definition:
(Identically distributed) Each sample $(x_i,y_i)$ is assumed to be sampled from a joint probability distribution $p(x_i,y_i)$ or in other words, each $(x_i, y_i)$ is the realization of the random variable $(X,Y)\sim p_{X,Y}(x_i,y_i)$
(Independent) For $i \neq j$, the realization $(x_i,y_i)$ is generated independently from $(x_j, y_j)$
Why I am unsatisfied/unsure about my definitions:
- For identically distributed, this question seems to say that each sample is drawn from the probability distribution over all the sample. That is, we assume that each data point $(x_i, y_i)$ is generated from a random variable $(X_i, Y_i)$, and that $(x_i,y_i)$ is not sampled from the distribution of $(X_i, Y_i) \sim p_{X,Y}(x_i,y_i)$ but from the distribution of all random variables $(X_1, \ldots, X_n, Y_1, \ldots, Y_n) \sim p(x_1, \ldots, x_n, y_1, \ldots, y_n)$ (subscript omitted). Which is different from the definition I have given above.
This is exactly what the notation $({\bf{X}}_i,y_i) \sim \mathbb{P}({\bf{X}},y), \forall i=1,...,N$ means in the linked question.
Which one is correct?
For independence: since $(x_i, y_i)$ and $(x_j,y_j)$ are just pairs of vectors, which are not random variables, hence we cannot speak about their independence. So to me, independence means that given $(x_i, y_i)$ and $(x_j,y_j)$ are generated by two (pairs of) random variables, $(X_i, Y_i)$ and $(X_j, Y_j)$, then the joint distribution $p_{X_i, X_j, Y_i, Y_j}(x_i, x_j, y_i, y_j)$ can be decomposed into $p_{X_i, Y_i}(x_i, y_i)p_{X_j, Y_j}(x_j, y_j)$.
Is this correct?
Just want to be mathematically rigorous about things. Any reference will help me!!!