2
$\begingroup$

I am reading the book 'Intro to probability and statistics using R' and in the chapter on linear regression, the author says:

enter image description here

Why does he say that the Y values are not identically distributed if he is saying that the Y values follow a normal distribution?

$\endgroup$
3
  • $\begingroup$ "Identical" implies the distributions will have the same mean. When $\beta_1 \ne 0$ andf $x_i \ne x_j$, do you suppose the means $\beta_0 + \beta_1 x_i$ and $\beta_0 + \beta_1 x_j$ will be the same? $\endgroup$
    – whuber
    Commented Oct 2, 2015 at 19:31
  • $\begingroup$ Thanks. I thought 'identical' means that they are both drawn from the normal distribution, didn't think that they need to be drawn from the same normal distribution $\endgroup$
    – Victor
    Commented Oct 2, 2015 at 19:50
  • 1
    $\begingroup$ That is a curious comment in that it uses "normal distribution" in two different senses within the same sentence! $\endgroup$
    – whuber
    Commented Oct 2, 2015 at 20:55

2 Answers 2

2
$\begingroup$

Identically distributed means that they have identical distributions, i.e. the same distribution function and the same parameters. Linear regression model in probabilistic terms is

$$ Y \sim \mathcal{N}(X\beta, \sigma^2) $$

In such model we can assume that $Y$'s are independent and exchangable, but not identically distributed, since they have different means

$$ E(Y\mid X,\beta) = \mu = X\beta $$

So unless you assume intercept-only model where there is a common mean for all $Y$, they are not i.i.d. What we can assume to be i.i.d., are the errors that follow the normal distribution with mean $0$ and variance $\sigma^2$.

$\endgroup$
0
$\begingroup$

This is an important question that touches on two points that cannot be reiterated enough:

  1. A probability distribution and a random variable are not the same thing
  2. We need to clear about what we assume and what is deduced

The book you are reading assumes that $\epsilon_{i}'s$ are i.i.d. $N(\mu=0,\sigma)$. In other words the assumption is that $\epsilon_1, \epsilon_2, ... , \epsilon_n$, which are different random variables, all have the same probability distribution. $\epsilon_1$ is the error random variable corresponding to the first observation, $\epsilon_2$ is the error random variable corresponding to the second observation, etc. It is key to notice that the only assumption the author makes is with regards to the distribution of the error terms. The distributions of the $Y_i$ are deduced or derived; not assumed. Note that the $Y_i$'s are random variables, because $Y_i=\beta_0+\beta_1x_i+\epsilon_i$, where $\epsilon_i$ is a random variable, thus whether the $x_i's$ are random variables or non-random variables, we can conclude that the $Y_i$'s are random variables. Note once again that $Y_1, Y_2, ... , Y_n$ are different random variables, and the question is whether all of these have the same probability distribution.

You can make this model come to life if you were to simulate it or at least imagine you simulated it. Let x be non-random and take value 1000 for observations 1-50, and value 2000 for 51-100. Suppose the unknown population parameters $\beta_0=2$ and $\beta_1=3$. For each observation of the 100, you will randomly generate an error value/residual from the $N(\mu=0,\sigma=1)$ distribution, having assumed a particular value for $\sigma$, we're now ready to see what our sample values of $Y_i$ will look like.

$Y_1=\beta_0+\beta_1x_1+\epsilon_1=2+3*1000+\epsilon_1=3002+\epsilon_1$ $Y_2=\beta_0+\beta_1x_2+\epsilon_2=2+3*1000+\epsilon_2=3002+\epsilon_2$

...

$Y_{50}=\beta_0+\beta_1x_{50}+\epsilon_{50}=2+3*1000+\epsilon_{50}=3002+\epsilon_{50}$ $Y_{51}=\beta_0+\beta_1x_{51}+\epsilon_{51}=2+3*2000+\epsilon_{51}=6002+\epsilon_{51}$

...

$Y_{100}=\beta_0+\beta_1x_{100}+\epsilon_{100}=2+3*2000+\epsilon_{100}=6002+\epsilon_{100}$

Every time we resample, we obtain 100 new values for the error terms, which imply new values for the $Y_i$'s. The values of $\epsilon_1, \epsilon_2, ... , \epsilon_{100}$ will differ from sample to sample and hence the values of $Y_1, Y_2, ... , Y_{100}$ will differ from sample to sample. Because we assumed the errors to follow $N(\mu=0,\sigma=1)$, we can conclude that: $E(Y_1)=...=E(Y_{50})=3002$ and $E(Y_{51})=...=E(Y_{100})=6002$, which is enough to conclude that $Y_1, Y_2, ... , Y_{100}$ will not all have the same probability distribution because they do not all have the same expected value. Finally, because $\epsilon_i$ ~ $N(\mu=0,\sigma=1)$:

$Y_{1,2,...,50}$ ~ $N(\mu=3002,\sigma=1)$ and $Y_{51,52,...,100}$ ~ $N(\mu=6002,\sigma=1)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.