What are the differences between stochastic and fixed regressors in linear regression model?

Question

If we have stochastic regressors, we are drawing random pairs $(y_i,\vec{x}_i)$ for a bunch of $i$, the so-called random sample, from a fixed but unknown probabilistic distribution $(y,\vec{x})$. Theoretically speaking, the random sample allows us to learn about or estimate some parameters of the distribution $(y,\vec{x})$.

If we have fixed regressors, theoretically speaking, we can only infer certain parameters about $k$ conditional distributions, $y\mid x_i$ for $i=1,2,\dots,k$ where each $x_i$ is not a random variable, or is fixed. More specifically, stochastic regressors allow us to estimate some parameters of the entire distribution of $(y,\vec{x})$ while fixed regressors only let us estimate certain parameters of the conditional distributions $(y,\vec{x_i})\mid x_i$.

The consequence is that fixed regressors cannot be generalized to the whole distribution. For example, if we only had $x=1,2,3,\dots,99$ in the sample as fixed regressors, we can not infer anything about $100$ or $99.9$, but stochastic regressors can.

This is a rather obscure question as many textbooks only talk about the differences in mathematical derivation but avoid discussing the differences to the extent they can be generalized theoretically.

Answer 1

My suggestion is to take the habit of calling the "fixed" regressors "deterministic". This accomplishes two things: first, it clears the not-infrequent misunderstanding that "fixed" means "invariant". Second, it clearly contrasts with "stochastic", and tells us that the regressors are decided upon (hence the "design matrix" terminology that comes from fields where the regressors are f... deterministic).

If regressors are deterministic, they have no distribution in the usual sense, so they have no moments in the usual sense, meaning in practice that $E(x^r) = x^r$. The only stochastic element in the sample, rests in the error term (and so in the dependent variable).

This has the basic implication that a sample with even one and varying deterministic regressor is no longer an identically distributed sample:

$$E(y_i) = bE(x_i) + E(u_i) \implies E(y_i) = bx_i$$

and since the deterministic $x_i$'s are varying, it follows that the dependent variable does not have the same expected value for all $i$'s. In other words, there is not one distribution, each $y_i$ has its own (possibly belonging to the same family, but with different parameters).

So you see it is not about conditional moments, the implications of deterministic regressors relate to the unconditional moments. For example, averaging the dependent variable here does not give us anything meaningful, except for descriptive statistics for the sample.

Reverse that to see the implication: if the $y_i$'s are draws from a population of identical random variables, in what sense, and with what validity are we going to link them with deterministic regressors? We can always regress a series of numbers on a matrix of other numbers: if we use ordinary least-squares, we will be estimating the related orthogonal projection. But this is devoid of any statistical meaning.

Note also that $E(y_i \mid x_i) = E(y_i)$. Does this mean that $y_i$ is "mean-independent" from $x_i$? No, this would be the interpretation if $x_i$ was stochastic. Here, it tells us that there is no distinction between unconditional and conditional moments, when deterministic regressors are involved.

We can certainly predict with deterministic regressors. $b$ is a common characteristic of all $y_i$'s, and we can recover it using deterministic regressors. Then we can take a regressor with a value out-of-sample, and predict the value of the corresponding $y$.

Answer 2

First, what is regression at all? See Definition and delimitation of regression model there is some disagreement about this very broad concept, but mostly it is about modeling the conditional distribution (or some aspect of it) of $Y$ given some predictors $x$.

So, given that we are going to condition on $x$, why should it matter at all if $x$ was random or deterministic at the beginning? See the similar question What is the difference between conditioning on regressors vs. treating them as fixed?.

I guess then that this random regressor thing seems such a mess because it really is a many-headed monster (somewhat like socialism, you cut one head and some other grow out.) So we must look at what could the reasons be for modelling the regressors as random. I try a short list, most surely not exhaustive:

1. Measurement errors in the regressors $x$. This could well occur even with designed experiments with deterministic regressors, so seems to me a separate problem. See the tags errors-in-variables or measurement-error.

2. Problems with data collection causing problems for inference, like regressors correlated with the error term, separate regressions with correlated error terms, and many other problems studied in econometrics and causal-inference, which cannot be modeled with deterministic regressors.

3. Models with lagged values of the response as a predictor. This is often done with regressors treated as deterministic, which seems strange to me. Then $Y$ is treated as random in one part of the model, and as deterministic in another part ...

It seems to me that this many cases is best treated on its own, and not under the very broad labeling as random regressors.

Answer 3

I don't think you describe the fixed regression correctly. The fixed in this context means that you can pick any level you decide.

Suppose, you're studying Web site outages as a function of parameters of the Web server and the load. Consider two different approaches:

• a. you do it in the load testing lab at your firm (in vitro)
• b. you do it on the live production server (in vivo)

A. In the load testing lab you can set any level of the load as well as any desired parameters of the Web server. You can load it with 1,000 simultaneous client and the worker pool size 100, and memory 100GB; or you could just have 10 simultaneous clients, 10 threads and 1GB etc.

In this case your fixed design matrix will have four columns: the intercept and three variables. It is fixed because there's nothing random about the variable levels. You know the exact values of each variable, and you chose them as you wished.

B. On the live production server, you can probably control only some parameters, and certainly can't control the load: clients will come and go as they wish. So, at least the load will be stochastic. Even the parameters are not completely fixed: after all you want the server still running and serving clients while you're testing it. Maybe you can play with the memory and thread pool settings in some ranges though. So, in the best case you can set only two variables out of three bona fide regressors.

You have the random design matrix in this case. You can only observe the load, which is the regressor here. This is a random variable.

Needless to say that analysis is much easier and more robust when you have a fixed design matrix.

Answer 4

I've upvoted a couple answers that already provide many of the ingredients to the answer. I'll provide what I view as a more direct answer.

Suppose you find a dataset with observations on 2 fields: x (fertilizer) and y (yield) but you don't know exactly how this dataset was obtained. You think of Fisher's experiments and realize that this is probably experimental data where x (amount of fertilizer) was set by the experimenter and after some time the corresponding y (crop yield) was measured. You want to fit the model $y=\beta_0+\beta_1x+\epsilon$.

What would it mean for you to treat x as non-random/fixed?

To treat x as non-random means to assume that the x was set by the experimenter in the precise sense that:

• $E(\epsilon|x)=E(\epsilon)$
• $Var(\epsilon|x)=Var(\epsilon)$

This is what most textbooks mean by a non-random regressor. Not only is x under the experimenter's control but it has been set in a particular way. For example, if the experimenter randomly pulled fertilizer amounts out of a hat, that would meet the above conditions. On the other hand, if the experimenter set fertilizer amounts as a function of plot quality, this would not meet the above conditions.

In this setting we would assume $E(\epsilon)=0$ and $Var(\epsilon)=\sigma^2$.

What would it mean for you to treat x as random?

To treat x as random means to assume that this is observational data where the x was merely observed and not set, which really says that we do not know the probability distribution of x.

In this setting we would assume $E(\epsilon|x)=0$ and $Var(\epsilon|x)=\sigma^2$.

Is there any other thing we could assume?

We could assume that x was set by the experimenter in a way that violated one of the above 2 conditions. This is still a non-random regressor in the dictionary sense as Var(x)=0 but this is in conflict with what textbooks mean by "non-random regressor". If the experimenter set fertilizer amounts as a function of plot quality then $E(\epsilon|x)\not=E(\epsilon)$ and even if we further assume that $E(\epsilon)=0$, note that $E(y|x)=\beta_0+\beta_1x+E(\epsilon|x)\not= \beta_0+\beta_1x+E(\epsilon)=\beta_0+\beta_1x$.

Answer 5

Many interesting things are already said but let me add some. First of all the asker is right that "stochastic vs non-stochastic" is badly treated point in literature and among specialists in general. Me too encountered situation like the asker told us, time ago.

It seems me that the problem come from two main points.

The first one is the meaning of “regression” as kjetil b halvorsen suggest us. Indeed, in general, several problems can come from unclear definitions. Today I convinced that regression must be intended as synonym of conditional expectation function (see here: Regression and the CEF ; Regression's population parameters ). Therefore something like “deterministic regressors” is an ambigous object because we cannot merge random and non random variables in a joint probability distribution. Sometimes “non stochastic regressors” is a bad terminology that stand for: we conditioning for $X$, so we can consider it as known, so as a constant (non stochastic). Indeed all quantities we are interested in (moments, distributions, estimates, ecc) are derived conditioning on $X$, no matters where $X$ come from. However the joint distribution ($y$,$X$) remain the, usually unknown, true starting point.

Just a note about variable like: constant, dummy, time trend, ecc. They are frequently used in regression and it seems me that we can use them (condition on them), even if they cannot be properly included in a joint probability distribution.

The second problem come from the debate between regression and causality. Indeed sometimes “non stochastic regressors” stand for something like “fixed in repeated samples” or “fixed by experimenter”. It seems me that Aksakal answer go in this direction. The experimental paradigm is common in econometrics. It bring us to the as if condition that permit us to achieve causal interpretation for regression. This way can work, but today I convinced that the structural causal paradigm is the best. I summarize my point of view here (Under which assumptions a regression can be interpreted causally?). Now, it seems me that an ambiguous concepts like “regressors fixed by experimenter” is useless and dangerous. Regression is regression, we cannot force this concept. If we want deal with interventions, we need another object. We need structural equations. In any case "non stochastic regressors" concept is surely not enough for proper causal inference (see here: non stochastic regressors and causation)

Finally, in some presentation most of the concept emerged in this discussion are used. However them remain deeply unclear. It seems me that an authoritative example is in the Greene widespread manual (8-th edition 2018 edition pag 25):