3
$\begingroup$

In the linear regression model we assume that the errors $ε_i$ are independent and identically distributed (i.i.d.) random variables. I am trying to understand what this assumption implies regarding the response variables $y_i$.

As far it concerns the identitally distributed assumption, since $y_i = X_ib+ε_i$ where $ε_i\sim \mathcal{N}(0,\,\sigma^{2})\,$ then $y_i\sim \mathcal{N}(Xb,\,\sigma^{2})\,$. So, the response variables are not identically distributed random variables because they do not have the same mean.

My questions are the following:

  1. Can we assume that $y|X=x$, for example $y|X=5$, are identically distributed since they follow the same distribution with the same mean and variance?
  2. Can we assume that $y$ are independent random variables or $y|X$ are independent random variables or neither of the two?
  3. In the machine learning context we assume that the data $(x_i,y_i)$ are i.i.d. What does this assumption implies about the random variable $y_i$. How is this related with the i.i.d. assumption of the errors in linear regression?
  4. Finally, regarding my opening statements that the errors are i.i.d. and $ε_i\sim \mathcal{N}(0,\,\sigma^{2})\,$ are they correct or we assume that the errors conditional on $X$ are i.i.d. and $ε_i|X_i\sim \mathcal{N}(0,\,\sigma^{2})\,$ or is the same thing?
$\endgroup$

1 Answer 1

7
$\begingroup$

Remember that the error terms in the regression measure the deviations of the response variable from its conditional mean (given knowledge of the explanatory variables). Indeed, under the stipulated model form for regression, this is essentially the definition of what the error terms are. So you have:

$$\varepsilon_i \equiv Y_i - \mathbb{E}(Y_i|X_i).$$

Observe that each error term is a function of both the response variable and explanatory variable for that data point. Now, if these values are IID then this means that the deviations-from-the-conditional-mean are independent and identically distributed. This does not lead to an IID response variable, except in the trivial case where the explanatory variable has zero variance (i.e., it has a point-mass distribution).

With regard to your specific questions, the answers are as follows:

  1. No. As you correctly point out, under the regression assumptions there is no common distribution for the response variable (and no common mean either). Other than in the trivial case where the explanatory variable has zero variance (i.e., it has a point-mass distribution) the response variable has a conditional mean that depends on the explanatory variable.

  2. The latter. The response variables are conditionally independent conditional on the explanatory variable. As a shorthand we may say that the values $Y_i|X_i$ are independent (though not identically distributed).

  3. This is a much stronger assumption than in regression analysis. It is equivalent to making the standard regression assumptions, but also assuming that the underlying explanatory variables are IID. Once you assume that $X_1,...,X_n \sim \text{IID}$, the regression assumptions imply that the response variable is also marginally IID, which gives the joint IID result.

  4. In regression analysis, all your distributional assumptions are conditional on the explanatory variables, so the actual assumption is that:

$$\varepsilon_1,...,\varepsilon_n | \mathbf{X} \sim \text{IID N}(0,\sigma^2).$$

$\endgroup$
5
  • $\begingroup$ thank you very much for your prompt response. I think there might is a misunderstanding regarding my first question, I understand why $y$ are not identically distributed since their mean depends on $X$ but I am asking whether $y|X$, for example $y|X=5$, are identically distributed since they have the same mean and variance $\endgroup$
    – gnikol
    Dec 4, 2021 at 22:06
  • 1
    $\begingroup$ The quantity $Y_i|X_i$ is a different thing to $Y_i|X_i=5$, so you will need to clarify --- perhaps you mean to ask if $Y_i|X_i=x$ are IID for a fixed $x$? $\endgroup$
    – Ben
    Dec 4, 2021 at 22:12
  • $\begingroup$ Yes thank you this is what I am asking. Does it hold? $\endgroup$
    – gnikol
    Dec 4, 2021 at 22:14
  • $\begingroup$ Yes, this does hold. (But note that it involves stipulating a common value for the explanatory variable for the data points, when this might not hold in the data.) $\endgroup$
    – Ben
    Dec 4, 2021 at 22:15
  • $\begingroup$ @gnikol Please do not use the "Your Answer" field to ask new or follow-up questions. Instead, click the blue ASK QUESTION link at the top of the page & ask there. $\endgroup$
    – Sycorax
    Dec 5, 2021 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.