8
$\begingroup$

We have a simple linear regression model. Our assumptions are:

$Y_i =\beta_0+\beta_1X_i+ \varepsilon_i $, $i=1, \cdots, n$

$\varepsilon_i \sim N(0, \sigma^2)$

$Var(\varepsilon_i|X_i=x)=\sigma^2$

$\varepsilon_1, \cdots, \varepsilon_n$ are mutually independent.

$\\$

Are these hypothesis enough to claim that $\varepsilon_i|X_i=x \sim N(0, \sigma^2)$?

$\endgroup$
10
  • $\begingroup$ In $\operatorname{Var}(\varepsilon_i|X=x)=\sigma^2$, $X$ does not have a subscript. Is that intentional? Also, in the very last formula $X$ does not have a subscript. Again, is that intentional? Are $X$ and $x$ vectors in both cases? $\endgroup$ Sep 9, 2015 at 17:53
  • $\begingroup$ @RichardHardy They are note vectors, the OP says "simple" linear regression, meaning only one explanatory variable, besides the constant term. $\endgroup$ Sep 9, 2015 at 18:03
  • $\begingroup$ @RichardHardy No, it was a mistake. $\endgroup$
    – DGRasines
    Sep 9, 2015 at 18:29
  • $\begingroup$ @AlecosPapadopoulos, I did notice that was a simple regression but then $X$ means a column vector and $X_i$ means one element of a vector. As the author noted, writing $X$ rather than $X_i$ was a mistake. $\endgroup$ Sep 9, 2015 at 18:34
  • 1
    $\begingroup$ Your title and your body questions don't seem to be asking the same thing. $\endgroup$
    – Glen_b
    Sep 9, 2015 at 23:28

2 Answers 2

13
$\begingroup$

No. Here's an interesting counterexample.

Define a density function

$$g(x) = \frac{2}{\sqrt{2\pi}}\exp(-x^2/2)I(-t \le x \le 0 \text{ or } t \le x)$$

for $t = \sqrt{2\log(2)} \approx 1.17741$. ($I$ is the indicator function.)

The plot of $g$ is shown here in blue. If we define $h(x) = g(-x)$, its plot appears in red.

Figure

Direct calculation shows that any variable $Y$ with density $g$ has zero mean and unit variance. By construction, an equal mixture of $Y$ with $-Y$ (whose PDF is $h$) has a density function proportional to $\exp(-x^2/2)$: that is, it is standard Normal (with zero mean and unit variance).

Let $X_i$ have a Bernoulli$(1/2)$ distribution. Suppose $\varepsilon_i|X=0$ has density $g$ and $\varepsilon_i|X=1$ has density $h$, with all the $(X_i, \varepsilon_i)$ independent. The assumption about $Y_i$ is irrelevant (or true by definition of $Y_i$) and all the other assumptions hold by construction, yet none of the conditional distributions $\varepsilon_i | X_i = x$ are Normal for any value of $x$.

Figures

These plots show a dataset of $300$ samples from a bivariate distribution where $E[Y|X]=5 + X.$ The $x$ values in the scatterplot at the left have been horizontally jittered (displaced randomly) to resolve overlaps. The dotted red line is the least squares fit to these data. The three histograms show the conditional residuals--which are expected to follow $g$ and $h$ closely--and then the combined residuals, which are expected to be approximately Normal.

$\endgroup$
2
  • 1
    $\begingroup$ (+1) This is the true value-added of CV. Thanks for this. $\endgroup$ Sep 10, 2015 at 1:20
  • 1
    $\begingroup$ The fun in math... So good! $\endgroup$ Sep 10, 2015 at 1:32
4
$\begingroup$

The assumption that the conditional variance is equal to the unconditional variance, together with the assumption that $E(\varepsilon_i)=0$, does imply zero conditional mean, namely

$$\{{\rm Var}(\varepsilon_i \mid X_i) = {\rm Var}(\varepsilon_i)\} \;\text {and}\;\{E(\varepsilon_i)=0\}\implies E(\varepsilon_i \mid X_i)=0 \tag{1}$$

The two assumptions imply that

$$E(\varepsilon_i^2 \mid X_i) -[E(\varepsilon_i \mid X_i]^2 = E(\varepsilon_i^2)$$ $$\implies E(\varepsilon_i^2 \mid X_i) - E(\varepsilon_i^2) = [E(\varepsilon_i \mid X_i]^2$$

Ad absurdum, assume that $E(\varepsilon_i \mid X_i)\neq 0 \implies [E(\varepsilon_i \mid X_i]^2 >0$

This in turn implies that $E(\varepsilon_i^2 \mid X_i) > E(\varepsilon_i^2)$. By the law of iterated expectations we have $E(\varepsilon_i^2) = E\big[ E(\varepsilon_i^2 \mid X_i)\big]$. For clarity set $Z \equiv E(\varepsilon_i^2 \mid X_i)$. Then we have that

$$E(\varepsilon_i \mid X_i)\neq 0 \implies Z > E(Z)$$

But this cannot be since a random variable cannot be strictly greater than its own expected value. So $(1)$ must hold.

Note that the reverse is not necessarily true.

As for providing an example to show that even if the above results hold, and even under the marginal normality assumption, the conditional distribution is not necessarily identical to the marginal (which would establish independence), whuber beat me to it.

$\endgroup$
12
  • 1
    $\begingroup$ Let us add an additional assumption that the true model is indeed linear and $\operatorname{E}(\varepsilon_i|X_i)=0$. We then have 1. unconditional normality, 2. conditional mean zero and 3. conditional constant variance. I still do not see whether (and if so, how) that yields conditional normality. (The sequence would work fine the other way around: conditional normality + conditional mean zero + conditional constant variance --> {unconditional normality+mean zero+constant variance.}) $\endgroup$ Sep 9, 2015 at 18:37
  • $\begingroup$ @RichardHardy Where in my answer do I write that conditional normality follows from the assumptions that you state? I explicitly write "conditional normality follows from nowhere". $\endgroup$ Sep 9, 2015 at 18:49
  • $\begingroup$ True. But you "blame" the fact that the true model may be nonlinear. If I may add this extra assumption about the true model and reiterate the question, would your answer change? $\endgroup$ Sep 9, 2015 at 18:54
  • $\begingroup$ @RichardHardy Where do I do that? The linearity/non-linerariy has to do with the mean-independence, not with conditional normality. $\endgroup$ Sep 9, 2015 at 18:56
  • 1
    $\begingroup$ Thanks, I thought so, too. However, I was unable to come up with a simple counterexample where all the conditions (plus the one stating that the true model is linear) are satisfied but the conditional distribution is nonnormal. A counterexample is always a nice way to disprove a hypothesis, so I thought it would be nice to come up with one. Anyway, yours is a nice answer. $\endgroup$ Sep 9, 2015 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.