Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
From page 8 of Rasmussen's book Gaussian Processes for Machine Learning:
This is all pretty basic...but what do they mean by "identically distributed" in the last sentence? I understand if we have a collection of random variables, we refer to them as identically distributed to mean they all have the same distribution. But here $\epsilon$ is just one random variable. So what does "identically distributed" mean?
You have $n$ observations, $y\in\mathbb{R}^n$. You correspondingly have $n$ noise terms, $\epsilon\in\mathbb{R}^n$. The last sentence means that each separate noise term $\epsilon_i$ is identically distributed and that they are independent.
(In more general situations, the $\epsilon_i$ may not be identically distributed, or dependent, e.g., in time series analysis. You need more complex tools in such a situation.)
Identically distributed generally means that each observation of a variable was sampled independently from a distribution identical to every other observation on that variable.
A simple random walk where $y_{y} = y_{t-1} + 2\mathcal{Bernouli}\left(0.5\right)-1$ is an example of a variable ($y_{t}$) that is not i.i.d.: each value of $y_{t}$ depends quite directly on it's immediately prior value, and in fact, the process "remembers" all perturbations to it infinitely. More, the variance of $y_{t}$ is a function of $t$ (in fact, $\sigma^{2}_{y_{t}}=t$), meaning that different values in this time series cannot possible have "the same" distribution.
