Sometimes we say the following: $X$ is some training data given by $X:=\{(x_1,y_1),...,(x_l,y_l)\}\subset R^d \text{x}R$. Assume that the training data had been drawn from independent and identical distribution (IID) from some probability distribution $P_{XY}$.

My question is that how can we assume IID when y is in fact dependent on x?


1 Answer 1


The $(x_i, y_i)$ are IID, but $x$ and $y$ are (in most applications) not IID.

Define $z_i \equiv (x_i, y_i)$. We usually assume that the $z_i$ are IID according to some multivariate distribution. That is very different from saying that $y_i$ is independent from $x_i$.

Here's a little R example:

n_obs <- 100
dataset <- data.frame(x=runif(n_obs))
dataset$y <- 5 + 3*dataset$x + rnorm(n_obs) 

The rows in that dataset are independent. Within each row, however, $x$ and $y$ are dependent: in this particular example, $(Y \;|\; X=x) \sim \mathcal{N}\left(5 + 3 \,x, \;1\right)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.