Simple Linear Regression Population (A Conceptual Question)

Question

I'm an applied researcher and had a conceptual question about the simple linear regression population.

Background

From some books, I know in simple linear regression population, we assume to have a "subpopulation" for each value of the predictor (see picture below). Each of these subpopulations, is assumed to be a Normal Distribution. All of these subpopulations are assumed to have the same $\sigma^2$. But each of these subpopulations may have a different $\mu$ provided that these differing $\mu$s can be perfectly linearly connected to each other (see my plot below where I show 4 subpopulations, all of which have the same $\sigma^2$ but with perfectly linearly related $\mu$s).

My Conceptual Question

So, each subpopulation is a normal distribution. BUT does that mean that the ENTIRE POPULATION (i.e., POPULATION = SUM of ALL Subpopulations; in the picture below, I'm showing 4 subpopulations of size $1000$, so suppose these 4 subpopulations together are my POPULATION of size $4000$) is also a normal distribution?

Answer 1

I understand your question to be, what's the distribution for the sum $a + bX + \epsilon$?

General case (if $X$ and $\epsilon$ are independent)

Let's say you have two random variables $X$ and $\epsilon$ with distribution functions $f_X$ and $f_\epsilon$ respectively. Assuming $X$ and $\epsilon$ are independent the distribution for the sum $Y = a + b X + \epsilon$ can be found by calculating the convolution:

$$ f_Y(y) = \int_{-\infty}^\infty f_{x}\left( \frac{y - a - t}{b} \right) f_\epsilon(t) dt$$

How that math works out will depend on the distributions given by $f_X$ and $f_\epsilon$.

Some intuition for above formula: If $\epsilon$ takes the value $t$, then you need $X$ to take the value $\frac{y - a - t}{b}$ for the sum of $X$ and $\epsilon$ to be $y$.

Some special cases:

1. $X$ and $\epsilon$ follow the normal distribution and are independent

• In some sense, this is the most well behaved, simple case.

• An equivalent assumption/situation is that the joint distribution over $X$ and $Y$ is multivariate normal.

If both $X$ and $\epsilon$ are normally distributed and independent, then $Y$ will be normally distributed. An important, classic result of probability theory is that the sum of two independent, normally distributed random variables also follows the normal distribution. (This can be shown by taking the convolution.)

2. $X$ takes discrete values: $P(X = x_i) = p_i$.

Let's say random variable $X$ takes the value $x_i$ with probability $p_i$, and that the error term $\epsilon \sim \mathcal{N}(0, \sigma^2)$.

Then distribution for $Y$ is a mixture of normals with probability density:

$$ f_Y(y) = \sum_i p_i f(y; x_i, \sigma^2)$$

where $f(y; \mu, \sigma^2)$ is the probability density for the normal distribution with mean $\mu$ and variance $\sigma^2$.