Conditional Mean in Linear Regression

Question

I have a question about linear regression in general. Suppose we have the following Data Generating Process:$$y_{i}=x_{i}\beta+\epsilon_{i}$$

Now, the thing is that from my understanding, each observation is a random variable, i.e. each $y_{i}$ has a conditional mean, that is:$$E[y_{i}|x_{i}]=x_{i}\beta$$ under exogeneity assumptions. My question is as follows. What is the unconditional mean of $y$? Also, when we have a sample of n data points, what really does $E[y]$ mean? Is it the mean of the random vector of $y_{i}'s$? How does the law of iterated expectations apply here?

Answer 1

It's important to be precise in these situations, and distinguish between the data model, and the data itself. One way to think about linear regression is that we hypothesize the following relationship on the unknowable statistical process that generated the data we do have

$$ E[Y \mid X] = \beta_0 + X \beta $$

Beta is an unknown constant at this point, so we are just setting down a hypothesis on what we believe the shape of the relationship is like. Then, given the data, we use some method to determine what $\beta$ should be so that the hypothesized relationship is likely to generate the data we do have (maximum likelihood being very popular).

Even without knowing $\beta$, we can manipulate the relationship to learn some things about the consequences of our assumptions

$$ E[Y] = E[E[Y \mid X]] = \beta_0 + \beta E[X] $$

Now, the distribution of $X$ is generally not part of our structural assumptions in regression, so, in general, this is as far as we can go.

Oftentimes, we will center our data for $X$, which imposes the constraint $E[X] = 0$ on our model. In this case, we can derive

$$ E[Y] = \beta_0 $$

This is why, for example this book recommends centering predictors (in some situations) so that the model intercept is interpretable.

Now, my question is how is this related to the sample average of y?

If you fit the model by least squares, and you have centered the predictor $x$, then the model intercept is the sample average.

Geometrically, the least squares line must pass through the center of mass of the data $(\bar x, \bar y)$. When you have centered $x$, $\bar x = 0$, so the line passes through $(0, \bar y)$. If you plug these values into the model equation, you get $\beta_0 = \bar y$.

Algebraically, the least squares equation is is $(X^t X) \vec{\beta} = X^t y$. If you think about the matrix $X$, the first column is all ones (the intercept column), and since $x$ is centered, this intercept column is orthogonal to the data column. This means that the first row of $X^t X$ looks like $(N, 0)$ (where $N$ is the number of data points). Then first component of the left hand side is $N\beta_0$. On the right hand side, the first component is $\sum_i y_i$. Equating them, you get the result $\beta_0 = \bar y$.

It is also true that the mean of the predictions is equal to $\bar y$. As these are the estimated conditional means (by assumption), this gives you a relationship like the one you seek. To see this, just observe that the predictions are $X \vec{\beta}$, and group the least squares equation as

$$ X^t (X \vec{\beta}) = X^t y $$

Now use a similar argument to what I did above.

Answer 2

To get the unconditional mean (or marginal mean) of Y, the distribution of X is needed when the mean of Y depends on X as in your question. If you do not know and cannot estimate the distribution of X, it is impossible to derive the unconditional mean of Y.