Consider the linear model $y_i = \beta^Tx_i + \epsilon_i$. If we assume $\epsilon_i$ are IID, then for MLE, it is claimed that $y_i|x_i$ is also IID for all $i$.
Does this always hold? If not, when does it break down?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ That should be true as long as none of $x_i$ predictors are lagged $y_i$. But, since you labelled them $x_i$, that's probably not the case. $\endgroup$– mloftonJul 19, 2020 at 2:36
-
$\begingroup$ @mlofton What do you mean by lagged $y_i$? Is that time series terminology where $x_i$ is some prior instance of $y$, e.g., $y_j$ such that j < i? $\endgroup$– DavidJul 19, 2020 at 3:40
-
1$\begingroup$ Hi David: Yes, exactly. Since you're referring to the non-time series formulation, you don't have to worry about it. my bad for lack of clarity there. I haven't read his answer carefully yet but , according to Ben, I'm wrong so make sure that you understand his answer. In my experience, it's probably correct :). $\endgroup$– mloftonJul 19, 2020 at 15:29
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
8
Marginal independence is a weaker condition than conditional independence, so the former does not imply the latter. If you are willing to make the stronger assumption that $\epsilon_1,...,\epsilon_n$ are IID conditional on the design matrix $\mathbf{x}$ (e.g., by assuming that the error terms are not only IID but also independent of the explanatory variables), then $y_1,...,y_n$ are also IID conditional on $\mathbf{x}$.
In regression analysis, we always proceed conditional on the explanatory variables, so the stronger assumption is the one that is made. (Usually this is an assumption that the error terms are IID and that they are jointly independent of the explanatory variables.) Often the conditioning statement is accidentally omitted in the statement of assumptions, and this is annoying becuase it leads to this kind of issue.
-
$\begingroup$ Is the stronger form of independence the usual assumption that we make in OLS regression? $\endgroup$– DaveJul 19, 2020 at 3:17
-
$\begingroup$ Somewhat related question that I have is in MLE with IID normally distributed errors, it states therefore that $p(y|x)$ is also normally distributed $\sim \mathcal{N}(x^T\beta, \sigma^2)$. I don't quite understand this. Why does it have to be conditional on $x$? Why can't it be the joint distribution of $x$ and $y$ that is normally distributed? $\endgroup$– DavidJul 19, 2020 at 3:47
-
$\begingroup$ @David: All regression results are conditional on the $x$ values, and we make no assumption about the distribution of those values. To see why this does not imply joint normality, consider what happens when $x_1,...,x_n$ are Bernoulli random variables (which does not breach any regression assumptions). How do you get joint normality then? $\endgroup$– BenJul 19, 2020 at 7:46
-
1$\begingroup$ @Ben: I see what you mean now by conditional independence but, if that assumption wasn't true ( the conditional independence assumption ), then the regression wouldn't meet the standard OLS assumptions. Still your point is well taken and I understand better now that there can be a difference between marginal and conditional independence.Thanks for clarification. $\endgroup$– mloftonJul 20, 2020 at 1:34
-
1$\begingroup$ gotcha. thanks for the thorough explanation. very insightful. $\endgroup$– mloftonJul 21, 2020 at 13:36