From page 8 of Rasmussen's book Gaussian Processes for Machine Learning:

enter image description here

This is all pretty basic...but what do they mean by "identically distributed" in the last sentence? I understand if we have a collection of random variables, we refer to them as identically distributed to mean they all have the same distribution. But here $\epsilon$ is just one random variable. So what does "identically distributed" mean?

  • $\begingroup$ This last sentence is definitely a sloppy formulation. There is no such thing as "identically distributed distribution". $\endgroup$
    – amoeba
    Commented Mar 2, 2018 at 23:03

2 Answers 2


You have $n$ observations, $y\in\mathbb{R}^n$. You correspondingly have $n$ noise terms, $\epsilon\in\mathbb{R}^n$. The last sentence means that each separate noise term $\epsilon_i$ is identically distributed and that they are independent.

(In more general situations, the $\epsilon_i$ may not be identically distributed, or dependent, e.g., in time series analysis. You need more complex tools in such a situation.)


Identically distributed generally means that each observation of a variable was sampled independently from a distribution identical to every other observation on that variable.

A simple random walk where $y_{y} = y_{t-1} + 2\mathcal{Bernouli}\left(0.5\right)-1$ is an example of a variable ($y_{t}$) that is not i.i.d.: each value of $y_{t}$ depends quite directly on it's immediately prior value, and in fact, the process "remembers" all perturbations to it infinitely. More, the variance of $y_{t}$ is a function of $t$ (in fact, $\sigma^{2}_{y_{t}}=t$), meaning that different values in this time series cannot possible have "the same" distribution.

  • $\begingroup$ Thank you for this great answer. Do I understand correctly that a collection of random variables can be identically distributed and the observations in a sample can be identically distributed, and in the first case identically distributed refers to random variables that have the same probability distribution, whereas in the second case it refers to the fact that each observation was sampled from the same distribution? $\endgroup$ Commented Feb 5, 2019 at 22:00
  • $\begingroup$ @ColorStatistics Um... yes? $\endgroup$
    – Alexis
    Commented Feb 6, 2019 at 0:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.