# Assume (x,y) are drawn from independent & identical distribution when y=f(x)

Sometimes we say the following: $X$ is some training data given by $X:=\{(x_1,y_1),...,(x_l,y_l)\}\subset R^d \text{x}R$. Assume that the training data had been drawn from independent and identical distribution (IID) from some probability distribution $P_{XY}$.

My question is that how can we assume IID when y is in fact dependent on x?

## 1 Answer

The $(x_i, y_i)$ are IID, but $x$ and $y$ are (in most applications) not IID.

Define $z_i \equiv (x_i, y_i)$. We usually assume that the $z_i$ are IID according to some multivariate distribution. That is very different from saying that $y_i$ is independent from $x_i$.

Here's a little R example:

n_obs <- 100
dataset <- data.frame(x=runif(n_obs))
dataset$y <- 5 + 3*dataset$x + rnorm(n_obs)


The rows in that dataset are independent. Within each row, however, $x$ and $y$ are dependent: in this particular example, $(Y \;|\; X=x) \sim \mathcal{N}\left(5 + 3 \,x, \;1\right)$.