6
$\begingroup$

From page 8 of Rasmussen's book Gaussian Processes for Machine Learning:

enter image description here

This is all pretty basic...but what do they mean by "identically distributed" in the last sentence? I understand if we have a collection of random variables, we refer to them as identically distributed to mean they all have the same distribution. But here $\epsilon$ is just one random variable. So what does "identically distributed" mean?

$\endgroup$
1
  • $\begingroup$ This last sentence is definitely a sloppy formulation. There is no such thing as "identically distributed distribution". $\endgroup$
    – amoeba
    Mar 2 '18 at 23:03
7
$\begingroup$

You have $n$ observations, $y\in\mathbb{R}^n$. You correspondingly have $n$ noise terms, $\epsilon\in\mathbb{R}^n$. The last sentence means that each separate noise term $\epsilon_i$ is identically distributed and that they are independent.

(In more general situations, the $\epsilon_i$ may not be identically distributed, or dependent, e.g., in time series analysis. You need more complex tools in such a situation.)

$\endgroup$
3
$\begingroup$

Identically distributed generally means that each observation of a variable was sampled independently from a distribution identical to every other observation on that variable.

A simple random walk where $y_{y} = y_{t-1} + 2\mathcal{Bernouli}\left(0.5\right)-1$ is an example of a variable ($y_{t}$) that is not i.i.d.: each value of $y_{t}$ depends quite directly on it's immediately prior value, and in fact, the process "remembers" all perturbations to it infinitely. More, the variance of $y_{t}$ is a function of $t$ (in fact, $\sigma^{2}_{y_{t}}=t$), meaning that different values in this time series cannot possible have "the same" distribution.

$\endgroup$
2
  • $\begingroup$ Thank you for this great answer. Do I understand correctly that a collection of random variables can be identically distributed and the observations in a sample can be identically distributed, and in the first case identically distributed refers to random variables that have the same probability distribution, whereas in the second case it refers to the fact that each observation was sampled from the same distribution? $\endgroup$ Feb 5 '19 at 22:00
  • $\begingroup$ @ColorStatistics Um... yes? $\endgroup$
    – Alexis
    Feb 6 '19 at 0:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.