7
$\begingroup$

I have a question about linear regression in general. Suppose we have the following Data Generating Process:$$y_{i}=x_{i}\beta+\epsilon_{i}$$

Now, the thing is that from my understanding, each observation is a random variable, i.e. each $y_{i}$ has a conditional mean, that is:$$E[y_{i}|x_{i}]=x_{i}\beta$$ under exogeneity assumptions. My question is as follows. What is the unconditional mean of $y$? Also, when we have a sample of n data points, what really does $E[y]$ mean? Is it the mean of the random vector of $y_{i}'s$? How does the law of iterated expectations apply here?

$\endgroup$
6
$\begingroup$

It's important to be precise in these situations, and distinguish between the data model, and the data itself. One way to think about linear regression is that we hypothesize the following relationship on the unknowable statistical process that generated the data we do have

$$ E[Y \mid X] = \beta_0 + X \beta $$

Beta is an unknown constant at this point, so we are just setting down a hypothesis on what we believe the shape of the relationship is like. Then, given the data, we use some method to determine what $\beta$ should be so that the hypothesized relationship is likely to generate the data we do have (maximum likelihood being very popular).

Even without knowing $\beta$, we can manipulate the relationship to learn some things about the consequences of our assumptions

$$ E[Y] = E[E[Y \mid X]] = \beta_0 + \beta E[X] = \beta_0 + \beta E[X] $$

Now, the distribution of $X$ is generally not part of our structural assumptions in regression, so, in general, this is as far as we can go.

Oftentimes, we will center our data for $X$, which imposes the constraint $E[X] = 0$ on our model. In this case, we can derive

$$ E[Y] = \beta_0 $$

This is why, for example this book recommends centering predictors (in some situations) so that the model intercept is interpretable.

Now, my question is how is this related to the sample average of y?

If you fit the model by least squares, and you have centered the predictor $x$, then the model intercept is the sample average.

Geometrically, the least squares line must pass through the center of mass of the data $(\bar x, \bar y)$. When you have centered $x$, $\bar x = 0$, so the line passes through $(0, \bar y)$. If you plug these values into the model equation, you get $\beta_0 = \bar y$.

Algebraically, the least squares equation is is $(X^t X) \vec{\beta} = X^t y$. If you think about the matrix $X$, the first column is all ones (the intercept column), and since $x$ is centered, this intercept column is orthogonal to the data column. This means that the first row of $X^t X$ looks like $(N, 0)$ (where $N$ is the number of data points). Then first component of the left hand side is $N\beta_0$. On the right hand side, the first component is $\sum_i y_i$. Equating them, you get the result $\beta_0 = \bar y$.

It is also true that the mean of the predictions is equal to $\bar y$. As these are the estimated conditional means (by assumption), this gives you a relationship like the one you seek. To see this, just observe that the predictions are $X \vec{\beta}$, and group the least squares equation as

$$ X^t (X \vec{\beta}) = X^t y $$

Now use a similar argument to what I did above.

$\endgroup$
  • $\begingroup$ Sorry @Matthew. I don't agree with your first equation; under exogeneity, the epsilon term should not be there. Another way to ask my question is can you make a statement about the conditional mean of the observation vs. the conditional mean of the dependent variable in the entire sample. Im guessing they are related! $\endgroup$ – ChinG Jan 27 '16 at 20:56
  • $\begingroup$ I believe you are absolutely correct on that first point, the $\epsilon$ should not be there, that's carelessness on my part, I'll fix it! I was trying to avoid the normality assumptions that are often stated here, but I got mixed up. Can you clarify your second point? $\endgroup$ – Matthew Drury Jan 27 '16 at 20:59
  • $\begingroup$ Thanks a lot for your quick answer. So my question is basically the following. Every observation has the conditional mean as you mentioned, which is basically x_i'b. Now, my question is how is this related to the sample average of y? I mean to compare the conditional mean of a specific observation with the unconditional mean across all observations in the sample. So basically, y_i would correspond to a specific observation, whereas Y would correspond to the vector of observations. My question is how is the conditional mean of each observation related to the mean across observations. Thanks! $\endgroup$ – ChinG Jan 27 '16 at 21:12
  • 1
    $\begingroup$ @ChinG I tried to address your question, hopefully that helps. $\endgroup$ – Matthew Drury Jan 28 '16 at 17:08
2
$\begingroup$

To get the unconditional mean (or marginal mean) of Y, the distribution of X is needed when the mean of Y depends on X as in your question. If you do not know and cannot estimate the distribution of X, it is impossible to derive the unconditional mean of Y.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.