I'm an applied researcher and had a conceptual question about the simple linear regression population.


From some books, I know in simple linear regression population, we assume to have a "subpopulation" for each value of the predictor (see picture below). Each of these subpopulations, is assumed to be a Normal Distribution. All of these subpopulations are assumed to have the same $\sigma^2$. But each of these subpopulations may have a different $\mu$ provided that these differing $\mu$s can be perfectly linearly connected to each other (see my plot below where I show 4 subpopulations, all of which have the same $\sigma^2$ but with perfectly linearly related $\mu$s).

My Conceptual Question

So, each subpopulation is a normal distribution. BUT does that mean that the ENTIRE POPULATION (i.e., POPULATION = SUM of ALL Subpopulations; in the picture below, I'm showing 4 subpopulations of size $1000$, so suppose these 4 subpopulations together are my POPULATION of size $4000$) is also a normal distribution?

enter image description here

  • 1
    $\begingroup$ In your attached imagine. It appears that your $x$ variables takes the discrete values 0, 1, 2, or 3. What you then have for the distribution of $y$ then is a mixture of normals which is not the same thing as as a normally distributed random variable. $\endgroup$ – Matthew Gunn May 18 '17 at 5:14
  • $\begingroup$ @MatthewGunn, I just did so for simplicity's sake (avoiding visual confusion), do please treat x (i.e., the predictor) as a continuous variable when answering my question. $\endgroup$ – rnorouzian May 18 '17 at 5:24
  • $\begingroup$ You cannot ignore @Matthew's example. It already shows the "population" distribution cannot be normal in general: you need to make a strong assumption about the distribution of $x$ for that to hold. $\endgroup$ – whuber May 18 '17 at 18:42
  • 1
    $\begingroup$ It, too, needs to be normal. It is rare that this is an important consideration. Your assumptions are far too stringent to be practicable. Most of the theory of least squares regression works, to an excellent approximation, when the conditional distributions are not normal (but are symmetric and not long tailed) and $x$ has practically any distribution. $\endgroup$ – whuber May 18 '17 at 19:02
  • 1
    $\begingroup$ What you do with simple linear regression is estimate a linear, conditional expectation function $\operatorname{E}[Y \mid X] = a + b X$. It turns out that's an interesting, sensible, and useful thing to do in a broad set of circumstances. $\endgroup$ – Matthew Gunn May 18 '17 at 19:37

I understand your question to be, what's the distribution for the sum $a + bX + \epsilon$?

General case (if $X$ and $\epsilon$ are independent)

Let's say you have two random variables $X$ and $\epsilon$ with distribution functions $f_X$ and $f_\epsilon$ respectively. Assuming $X$ and $\epsilon$ are independent the distribution for the sum $Y = a + b X + \epsilon$ can be found by calculating the convolution:

$$ f_Y(y) = \int_{-\infty}^\infty f_{x}\left( \frac{y - a - t}{b} \right) f_\epsilon(t) dt$$

How that math works out will depend on the distributions given by $f_X$ and $f_\epsilon$.

Some intuition for above formula: If $\epsilon$ takes the value $t$, then you need $X$ to take the value $\frac{y - a - t}{b}$ for the sum of $X$ and $\epsilon$ to be $y$.

Some special cases:

1. $X$ and $\epsilon$ follow the normal distribution and are independent

  • In some sense, this is the most well behaved, simple case.

  • An equivalent assumption/situation is that the joint distribution over $X$ and $Y$ is multivariate normal.

If both $X$ and $\epsilon$ are normally distributed and independent, then $Y$ will be normally distributed. An important, classic result of probability theory is that the sum of two independent, normally distributed random variables also follows the normal distribution. (This can be shown by taking the convolution.)

2. $X$ takes discrete values: $P(X = x_i) = p_i$.

Let's say random variable $X$ takes the value $x_i$ with probability $p_i$, and that the error term $\epsilon \sim \mathcal{N}(0, \sigma^2)$.

Then distribution for $Y$ is a mixture of normals with probability density:

$$ f_Y(y) = \sum_i p_i f(y; x_i, \sigma^2)$$

where $f(y; \mu, \sigma^2)$ is the probability density for the normal distribution with mean $\mu$ and variance $\sigma^2$.

  • $\begingroup$ Matthew, is what you are saying the basis of simple linear regression? assumptions as I describe it in my question? $\endgroup$ – rnorouzian May 18 '17 at 5:52
  • $\begingroup$ @parvinkarimi You want to know the distribution of $y$, correct? That is the sum of two random variables $x$ and $\epsilon$. The distribution of a sum is found by taking the convolution. For the most simple case of linear regression, the assumption is that the distribution of the error term $\epsilon$ conditioning on x is normal. Or another possible assumption is that $x$ isn't a random variable (eg. all the values are chosen by the person conducting the experiment). $\endgroup$ – Matthew Gunn May 18 '17 at 5:57
  • $\begingroup$ So, My question is solely focused on the simple linear regression assumptions as I indicated in my question. $\endgroup$ – rnorouzian May 18 '17 at 6:00
  • $\begingroup$ @parvinkarimi If $X$ is normally distributed, $\epsilon$ is normally distributed, and they are independent, then $Y$ will be normally distributed. That's probably the case you want. Note though that if $X$ only takes the values 0, 1, 2, or 3, then that's not the case you're in. $\endgroup$ – Matthew Gunn May 18 '17 at 6:09
  • $\begingroup$ Matthew - your comment mentions mixtures of normals but your answer doesn't. It would be good to make reference to that in your answer. $\endgroup$ – Glen_b May 18 '17 at 8:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.