What is a random variable and what isn't in regression models

Question

I've already seen this question but it didn't help .

So I'm going over regression models (simple linear regression mainly) in my statistics text book and there's a lot of confusion here about what actually is a random variable and what isn't. Namely, at one point they treat some term as a random variable and then later it's a constant. Or something is initially a constant but then we calculate its expected value somehow.

Anyway we first define regression function as $f(X) = E(Y|X)$, after which we immediately go specifically to simple linear regression.

Let $(X_1, Y_1), ... (X_n, Y_n)$ be our sample. The model that we wish to apply is $$Y_i = \beta_0 + \beta_1X_i + \epsilon_i$$ where the sequence of random variables $\{\epsilon_i\}$ satisfies the following:

1. $E(\epsilon_i) = 0 $ for $i=1, 2, ..., n$
2. $E(\epsilon_i\epsilon_j) = 0$ for all $i \neq j$
3. $D(\epsilon_i)=\sigma^2 < \infty$

The problem with this textbook is that everything is very vague and it's written as if it's supposed to be a reminder for someone who already knows all this stuff rather then a textbook for someone to learn it from scratch from.

Later on we derive the estimated coefficients $\beta_0$ and $\beta_1$ using partial derivatives of the sum of squares, and we obtain:

$$\hat{\beta_1} = \frac{\sum_{i=1}^n(X_i - \bar{X_n})(Y_i-\bar{Y_n})}{\sum_{i=1}^n(X_i-\bar{X_n})^2}$$ $$\hat{\beta_0} = \bar{Y_n} - \hat{\beta_1}\bar{X_n}$$

Now we wish to find the expected value for $\hat{\beta_1}$. We transform it into the following form: $$\hat{\beta_1} = \sum_{i=1}^n{Y_i\frac{(X_i - \bar{X_n})}{nS^2_{X}}}$$ where $S^2_{X}$ is $\frac{1}{n}\sum_{i=1}^n(X_i - \bar{X_n})^2$.

And now when we start finding the expected value it looks something like this:

$$E(\hat{\beta_1}) = \sum_{i=1}^n{E(Y_i)\frac{X_i - \bar{X_n}}{nS^2_{X}}} = \sum_{i=1}^n{(\beta_0 + \beta_iX_i)\frac{X_i-\bar{X_n}}{nS^2_{X}}} = ...$$

Meaning, everything except for $Y_i$ in the sum is treated as a constant. That's one of the parts I don't understand. In some other sources where I've tried finding answers to this question I've seen the following sentence:

Only ${e_i}$'s are random variables

This doesn't sit right with me probably because I got to regression after I'd been studying hypothesis testing and other parts of statistical inference for a while, where we've always treated 'almost everything' as a random variable, meaning the sample (in this case the $X_i, Y_i$ pairs), was also a random variable. How come here, suddenly, the part containing $X_i$ and $\bar{X_n}$ gets just thrown out of the $E()$ as if it is just a constant?

Some sources also mention that $X_i, Y_i$'s are indeed random variables but rather 'fixed', which still doesn't help me understand it because it sounds very informal.

Now I'll try and summarize my question(s) somehow.

1. Do we treat $(X_i, Y_i)$'s as random variables?
2. Do we treat $\beta_0$ and $\beta_1$ as random variables?
3. Do we treat $\hat{\beta_0}$ and $\hat{\beta_1}$ as random variables?
4. What can have an expected value and what can't (what gets treated as a constant when finding expected values) and why?

Answer 1

This post is an honest response to a common problem in the textbook presentation of regression, namely, the issue of what is random or fixed. Regression textbooks typically blithely state that the $X$ variables are fixed and go on their merry way, when in practice this assumption eliminates most of the interesting regression applications.

Rather than assume the $X$ variables are fixed, a better route to understanding regression analysis is to take a conditional distribution approach, one where the $X$'s are assumed random throughout, and then the case of fixed $X$ (which occurs only in very narrow experimental designs, and at that only when the experiment is performed without error) is subsumed as a special case where the distributions are degenerate.

What the OP is missing is the link from random $X$ to fixed realizations of $X$ ($X=x$), which all starts from the

Law of Total Expectation: Assume $U$ and $V$ are random, with finite expectation. Let $E(U | V=v) = \mu(v)$. Then $E(U) = E\{\mu(V)\}$.

This "Law" (which is actually a mathematical theorem) allows you to prove unbiasedness of the estimate $\hat \beta $ in two steps: (i) by first showing that it is unbiased, conditional on the $X$ data, and (ii) by using the Law of Total Expectation to then show that it is unbiased when averaged over all possible realizations of the $X$ data. (The average of 11,11, 11, 11, 11, 11, ... is 11, e.g.).

Answers to the OP:

Q1. Do we treat $(X_i,Y_i)$'s as random variables?

A1. Yes. They are random in the sense of the model, which describes the way that potentially observable values of such data might appear. Of course the actual observed data, $(x_i, y_i)$, are not random. Instead, they are fixed values, one many possible realizations of the potentially observable random variables $(X_i, Y_i)$. In rare cases, the $X$ data are fixed, but this is covered as a special case of randomness, so it is easier and safer just to assume randomness always.

Q2. Do we treat $\beta_0$ and $\beta_1$ as random variables?

A2. This is somewhat off topic from the OP, but still a very important question. From the scientist's conceptualization of reality, these are ordinarily fixed values. That is, the scientist assumes that there is a rigid structure responsible for the production of all of the $(Y_i | X_i = x_i)$ data values, and these $\beta_0, \beta_1$ values are part of that rigid structure.

Now, the parameters $\beta_0, \beta_1$ are uncertain in the scientist's mind (which is why he or she is collecting data in the first place!), so the scientist may choose to view them, mentally, as "random." The scientist has some ideas about the possible values of these parameters based on logic, subject matter considerations, and past data, and these ideas form the scientist's "prior distribution." The scientist then may update this prior using current data to obtain her/his posterior. That, in a nutshell, in what Bayesian statistics is all about.

But again, that issue is a little off topic from the OP, so let's consider everything conditional on the scientist's conceptualization that there is a rigid structure, and that these $\beta_0, \beta_1$ values are fixed in reality. In other words, all of my replies other than this one assume that the $\beta$'s are fixed.

Q3. Do we treat $\hat \beta_0$ and $\hat \beta_1$ as random variables?

A3. Here is another place where typical regression teaching sources are slippery. In some cases, they refer to the estimates $\hat \beta_0$ and $\hat \beta_1$ as functions of the (fixed) data that has been collected, and sometimes they refer to them as functions of the (random) potentially observable data, but use the same symbols $\hat \beta_0$ and $\hat \beta_1$ in either case. Often, you just have to understand from context which is which.

Whenever you see $E(\hat \beta)$, you can assume that $\hat \beta$ is a function of the random data, i.e., that $\hat \beta$ is a function of the $(X_i, Y_i)$.

Whenever you see the value of $\hat \beta$ reported, e.g., following a computer printout of results from a regression analysis, you can assume that $\hat \beta$ is a function of the fixed data sample, i.e., that $\hat \beta$ is a function of the $(x_i, y_i)$.

Q4. What can have an expected value and what can't (what gets treated as a constant when finding expected values) and why?

A4. Anything can have an expectation. Some things are more interesting than others, though. Anything that is a fixed (like a $\hat \beta$ that is a function of the observed $(x_i, y_i)$ sample) has an expectation that is just equal to that value. For example, if you observe from your computer printout that $\hat \beta_1 =0.23$, then $E(\hat \beta_1) =0.23$. But that is not interesting.

What is more interesting is the following question: over all possible potential realizations of $(X_i, Y_i)$ from this data-generating process, is the estimator $\hat \beta_1$ neither systematically too large, nor systematically too small, in an average sense, when compared to the structural parameter $\beta_1$? The expression $E(\hat \beta_1) = \beta_1$ tells you that the answer to that question is a comforting "yes."

And in that expression $E(\hat \beta_1) = \beta_1$, it is implicit that $ \hat \beta_1$ is a function of the potentially observable $(X_i, Y_i)$ data, not the sample $(x_i, y_i)$ data.

Answer 2

Do we treat $(X_i,Y_i)$'s as random variables?

In a regression model $Y=X\beta+\epsilon$, $\epsilon$ is a random variable and therefore $Y$, a transformation of a random variable, is itself a random variable.

The explanatory variables may be random or fixed. Tipically they are fixed when the researcher "controls" or "sets" the values of the explanatory variables. In experimental studies "the individuals or material investigated, the nature of the treatments or manupulations under study and the measurement procedure used are all selected, in their important features at least, by the investigator" (Cox and Reid, The Theory of the Design of Experiments, CRC, 2000, p. 1). For example, in a clinical study drugs and their doses are decided by the researcher, are fixed and known quantities, not random variables.

However, one can also think of stratified sampling, with the values of $X$ defining the strata, or subpopulations. "For example, if $X$ denotes gender, a researcher may decide to collect a sample consisting of 50 men, followed by 25 women. If so, the sample values of $X$ are nonstochastic as required, but the researcher has not controlled, set, or manipulated the gender of any individual in the population" (Arthur Goldberger, A Course in Econometrics, Harvard University Press, 1991, p. 148). In stratified sampling $X$ may be random, but $n$ values are specified, they define $n$ subpopulations, and are mantained in repeated sampling, so the expectation of each $Y_i$ will depend only on $i$ (Goldberger, p. 172).

In random sampling from a multivariate population both $Y$ and $X$ are random variables. This often happens in observational studies, where the researcher observes several subjects, measures several variables together, looks for their joint dependence. A typical example is econometrics (Bruce Hansen, Econometrics, §1.4).

Do we treat $\beta_0$ and $\beta_1$ as random variables?

In "classical" statistical inference, parameters are just unknown quantities. (In bayesian inference parameters are random variables.)

Do we treat $\hat\beta_0$ and $\hat\beta_1$ as random variables?

In "classical" statistical inference estimators are random variables.

What can have an expected value and what can't (what gets treated as a constant when finding expected values) and why?

If $X$ is nonrandom, then you assume $E[\epsilon]=0$ and look for $E[Y]=X\beta$. If $X$ is random, then you also assume $E[\epsilon\mid X]=0$, and look for $E[Y\mid X]$.

Answer 3

First note that minimizing the least squares sum $$Q(\beta) = \sum_{i=1}^n (f_\beta(x_i) - y_i)^2$$ is a generlal principle that can be applied independent from the actually underlying model. It can be shown, however, that this principle is equivalent to the maximum likelihood solution for a particular statistical model:

1. $x_1,\ldots,x_i$ are assumed to be exact
2. $y_i$ are assumed to be random variables subject to $y_i=f_\beta(x_i) + \epsilon_i$ where $\epsilon_i$ is a normally distributed random variable with mean zero and an unknown variance $\sigma^2$
3. the function parameters $\beta=(\beta_1,\ldots,\beta_k)$ have a constant, but unknown value

Thus, $x_i$ and $\beta$ are constants, and $y_i$ are (mutually independent) random variables. The estimators $\hat{\beta}_1,\ldots$ are random variables because they depend on the (random) values for $y_i$.

One remark: what I found very confusing when learning linear regression was that it is often called "least squares fitting", which seems to imply that ordinary least squares (OLS) is about fitting a curve to data. This makes the first assumption, however, very unrealistic, because in practice both $y_i$ and $x_i$ have measurement errors and are both random. It took me some time to understand that OLS is not about fitting, but about prediction. The question is: what is the value of $Y$ for a given $X$.

Answer 4

Do we treat $(X_i,Y_i)$'s as random variables?

we do treat $Y_i$ as a random variables in a sense that they are created by the underlying data generating process which we can't directly observe even if there is some non-random $y(x)$. For example, even if the underlying relationship is very close to being linear $y(x) \approx \beta_0 +\beta_1 x$ but not exact we will have to add there random 'disturbance' term $y(x) = \beta_0 +\beta_1 x + u$. Also please note this is not the OLS specification this is the assumption about the underlying relationship. So even if $x$ is treated as given $y$ should be treated as random variable.

Do we treat $\beta_0$ and $\beta_1$ as random variables?

These are treated as given - they are the unknown constants of the relationship you are trying to model so these are not random. However, we cannot observe what these variables are because we cant directly observe what the function that generates data we observe is. We can only estimate it.

Do we treat $\hat{\beta}_0$ and $\hat{\beta}_1$ as random variables?

Yes because as mentioned in the other answer you linked to your question these depends on what the $Y_i$ are. However, note that once you collect some given sample of $Y_i$ then within the collected sample $\hat{\beta}_0$ and $\hat{\beta}_1 $ wont be random - if you run same regression on same sample even fifty times you still get exactly the same estimated values for betas. However, they are random in a sense that $Y_i$ in our sample are randomly generated - even though once you finish collecting your sample $Y_i$ values are fixed. However, point of econometrics is not to just make conclusions about your sample but to generalize them and to do that you have to treat your sample as collection of random points generated by the data generating process. Otherwise if you would want to do just inferences within the sample and not any general ones there would not be any point in even testing significance of coefficients because inside the sample they hold.

What can have an expected value and what can't (what gets treated as a constant when finding expected values) and why?

Expectations are not indicators of what is and what isn't random. You can take an expectation of a constant, random variable or even combination of them. For example $E[c]= c$ where $c$ is some arbitrary constant.