In this question, it was stated that the assumption of i.i.d. for data comes in the form of $$(X_i,y_i)∼P(X,y),∀i=1,...,N \\(X_i,y_i) \;independent\; of \;(X_j,y_j),\;∀i≠j∈{1,...,N} $$ I am clear with the definition of i.i.d. and its concepts, however it is still rather unclear to me how this assumption is applicable.

To illustrate my confusion with an example, say we are looking at a classification problem, where $X$ is the input feature and $y$ is the label.

When we generate $n$ samples for training, I would think of it as drawing $(X_i,y_i)$ from the joint distribution of $X$ and $y$. How is the concept of independent and identical distribution relevant here then? Aren't $(X_i,y_i),\; for \; i =0,...,n$ all being drawing from the same distribution of $X$ and $y$.

  • $\begingroup$ The related answer might also help. $\endgroup$
    – Ben
    Commented Sep 30, 2021 at 10:14

1 Answer 1


The details are discussed in the On the importance of the i.i.d. assumption in statistical learning thread, but answering your question: the $n$ samples your observed are considered as random variables. So all the $(X_i, y_i)$ pairs are thought as $n$ random variables. Only random variables can be independent or have probability distributions, so if they are "independent and identically distributed", we must be talking about random variables. To be able to think about your data in probabilistic terms, you need to think of them as random variables.

  • $\begingroup$ The main part I am confused is that, say we are doing an image classification task for animals, so $X$ would be the input feature and therefore all possible image vector that contain animals, for each training data, isn't it just a sample from this population? Why do we need to consider each sample as a random variable on its own? $\endgroup$
    – tangolin
    Commented Sep 30, 2021 at 7:02
  • 1
    $\begingroup$ @tangolin because otherwise you cannot build a probabilistic model for it. If those are "just numbers" they don't have probability distributions etc. You need to think of them as random variables so you can put them into the probabilistic framework. $\endgroup$
    – Tim
    Commented Sep 30, 2021 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.