22
$\begingroup$

Sometimes we assume that regressors are fixed, i.e. they are non-stochastic. I think that means all our predictors, parameter estimates etc. are unconditional then, right? Might I even go so far that they are no longer random variables?

If on the other hand we accept that most regressors in economics say are stochastic because no outside force determined them with some experiment in mind. Econometricians then condition on these stochastic regressors.

How is this different from treating them as fixed?

I understand what conditioning is. Mathematically, it means we make all observations and inference conditional on that particular set of regressors and have no ambitions to say that inferences, parameter estimates, variance estimates etc. would have been the same had we seen a different realization of our regressors (such is the crux in time series, where each time series is only ever seen once).

However, to really grasp the difference between fixed regressors vs. conditioning on stochastic regressors, I am wondering if anyone here knows of an example of an estimation or inference procedure that is valid for say fixed regressors but breaks down when they are stochastic (and will be conditioned on).

I am looking forward to seeing those examples!

$\endgroup$
5
  • 2
    $\begingroup$ Are you familiar with errors-in-variables models? $\endgroup$ Commented Apr 8, 2015 at 3:23
  • $\begingroup$ Hey @robin.datadrivers no I am not actually. $\endgroup$
    – Hirek
    Commented Apr 8, 2015 at 9:35
  • 1
    $\begingroup$ These are models specifically designed to adjust estimates for measurement error in the independent variables. Not quite the same as stochastic regressors, but it could be useful for you to take a look. Also, survey research in general often assumes independent variables collected by surveys have sampling error - there are probably models out there that account for sampling error. $\endgroup$ Commented Apr 13, 2015 at 18:40
  • 1
    $\begingroup$ Another thought I came across was to use Bayesian models. Bayesian models can treat regressors as random, by specifying a prior distribution for them. Typically if they are treated as fixed, you specify a prior distribution only for the parameters (coefficients, means, variances), but when you have missing covariates or outcomes, you specify a prior distribution for them. I don't know exactly how I would implement it without more thought, but maybe there is a way to specify a prior distribution for each independent variable. $\endgroup$ Commented Apr 20, 2015 at 14:08
  • $\begingroup$ Related: stats.stackexchange.com/questions/215230/… ; stats.stackexchange.com/questions/173660/… $\endgroup$
    – markowitz
    Commented Nov 15, 2023 at 16:53

2 Answers 2

16
$\begingroup$

Here I am on thin ice but let me try: I have a feeling (please comment!) that a main difference between statistics and econometrics is that in statistics we tend to consider the regressors as fixed, hence the terminology design matrix which obviously comes from design of experiments, where the supposition is that we are first choosing and then fixing the explanatory variables.

But for most data sets, most situations, this is a bad fit. We are really observing the explanatory variables, and in that sense they stand at the same footing as the response variables, they are both determined by some random process outside our control. By considering the $x$'s as "fixed", we decide not to consider a lot of problems which that might cause.

By considering the regressors as stochastic, on the other hand, as econometricians tend to do, we open the possibility of modeling which try to consider such problems. A short list of problems we then might consider, and incorporate into the modeling, is:

Probably, that should be done much more frequently that it is done today? Another point of view is that models are only approximations and inference should admit that. The very interesting paper The Conspiracy of Random Predictors and Model Violations against Classical Inference in Regression by A. Buja et.al. takes this point of view and argues that nonlinearities (not modeled explicitely) destroys the ancillarity argument given below.

EDIT 

I will try to flesh out an argument for conditioning on regressors somewhat more formally. Let $(Y,X)$ be a random vector, and interest is in regression $Y$ on $X$, where regression is taken to mean the conditional expectation of $Y$ on $X$. Under multinormal assumptions that will be a linear function, but our arguments do not depend on that. We start with factoring the joint density in the usual way $$ f(y,x) = f(y\mid x) f(x) $$ but those functions are not known so we use a parameterized model $$ f(y,x; \theta, \psi)=f_\theta(y \mid x) f_\psi(x) $$ where $\theta$ parameterizes the conditional distribution and $\psi$ the marginal distribution of $X$. In the normal linear model we can have $\theta=(\beta, \sigma^2)$ but that is not assumed. The full parameter space of $(\theta,\psi)$ is $\Theta \times \Psi$, a Cartesian product, and the two parameters have no part in common.

This can be interpreted as a factorization of the statistical experiment, (or of the data generation process, DGP), first $X$ is generated according to $f_\psi(x)$, and as a second step, $Y$ is generated according to the conditional density $f_\theta(y \mid X=x)$. Note that the first step does not use any knowledge about $\theta$, that enters only in the second step. The statistic $X$ is ancillary for $\theta$, see https://en.wikipedia.org/wiki/Ancillary_statistic.

But, depending on the results of the first step, the second step could be more or less informative about $\theta$. If the distribution given by $f_\psi(x)$ have very low variance, say, the observed $x$'s will be concentrated in a small region, so it will be more difficult to estimate $\theta$. So, the first part of this two-step experiment determines the precision with which $\theta$ can be estimated. Therefore it is natural to condition on $X=x$ in inference about the regression parameters. That is the conditionality argument, and the outline above makes clear its assumptions.

In designed experiments its assumption will mostly hold, often with observational data not. Some examples of problems will be: regression with lagged responses as predictors. Conditioning on the predictors in this case will also condition on the response! (I will add more examples).

One book which discusses this problems in a lot of detail is Information and exponential families: In statistical theory by O. E Barndorff-Nielsen. See especially chapter 4. The author says the separation logic in this situation is however seldom explicated but gives the following references: R A Fisher (1956) Statistical Methods and Scientific Inference $\S 4.3$ and Sverdrup (1966) The present state of the decision theory and the Neyman-Pearson theory.

$\endgroup$
1
  • 1
    $\begingroup$ (+1) I wish I read this answer years ago. $\endgroup$
    – rando
    Commented May 8, 2023 at 19:46
5
$\begingroup$

+1 to Kjetil b halvorsen. His answers are enlightening and this one is no exception. I do think that there is something additional to be contributed here because the question asks about "treating regressors as fixed" (as in a hypothetical intervention to use Pearl's language) but also touches on "fixing the regressors" (as in a real design experiment).

enter image description here


This is where it gets confusing. Let's distinguish between 3 different paradigms:

  1. You design an experiment. You will set the level of fertilizer to either 1, 2, 3 units (this is the regressor) and then observe the yield (this is the outcome variable). This is a REAL experiment. You performed it. The regressor in this case is non-random because you determined how much fertilizer to put on each plot and not the roll of a dice or some other random experiment.
  2. You have an observational dataset on yield and fertilizer and you are not sure how yield was assigned to the plots, so you cannot assume that it was assigned randomly. You are interested in $E[$yield|fertilizer$=3]-E[$yield|fertilizer$=2]$. This amounts to a filtering of your dataset to the plots that were assigned 3 units of fertilizer and calculating their average yield then filtering the dataset to the plots that were assigned 2 units of fertilizer and calculating their average yield and then take the difference of the 2 averages. In this case conditioning amounts to filtering. It is key to note that this is not the causal effect of increasing fertilizer from 2 to 3. It is just a summary of your existing dataset.
  3. You have an observational dataset on yield and fertilizer and you do know that plots in sunnier areas were applied more fertilizer and your knowledge of agriculture tells you that more sun translates into a higher yield. Suppose that nothing else determined jointly both how the fertilizer was assigned and the outcome, so that you can assume that your causal DAG is complete and correct. Suppose you are interested in the causal effect of fertilizer on yield when the amount of fertilizer was increased from 2 to 3. Using Judea Pearl's do operator this question can be equivalently written as: $$E[yield|do(fertilizer=3)]-E[yield|do(fertilizer=2)]$$ In words, this question asks for the difference in the average yield if we performed a hypothetical experiment in which we first assigned every plot 2 units of fertilizer and computed the average yield, then applied every plot 3 units of fertilizer and computer the average yield and then took the difference between these 2 averages. To answer this question we'll have to condition Y=yield on both X=fertilizer and Z=sunniness of the plot.

In the 3rd case, you are imagining an alternative world, different that reality; you are imagining something counterfactual. This is where you imagine a world in which the level of the regressor had been fixed to a particular value. In the 2nd case, you accept/observe the reality as is and want to summarize it. The regressor is random and you condition on it to get a summary of your filtered dataset. In the 1st case you create the reality. You fix the regressors in the real world and will have to get some dust on your boots as well because you are actually performing the experiment.

Sometimes we assume that regressors are fixed, i.e. they are non-stochastic. I think that means all our predictors, parameter estimates etc. are unconditional then, right? Might I even go so far that they are no longer random variables?


That is not quite correct. When regressors are deterministic/non-random, yes they are not random variables. However, the OLS estimators are still very much random variables because they are linear combinations of $Y_i$, and the $Y_i$ are random variables (even if all regressors are deterministic) because $\epsilon_i$ are random variables. Yes, when x is non-random: $$E[Y|x]=\beta_0+\beta_1x+E[\epsilon|x]=\beta_0+\beta_1x+E[\epsilon]=E[Y]$$ but when X is random: $$E[Y|X]=\beta_0+\beta_1X+E[\epsilon|X]\not=\beta_0+\beta_1E[X]+E[\epsilon]=E[Y]$$ This is a key difference.


$\endgroup$
2
  • $\begingroup$ great answer! I would like to ask: why in the last paragraph in the case $x$ is fixed the equality holds but with $X$ random it doesn't? $\endgroup$
    – Tortar
    Commented Feb 20, 2022 at 1:35
  • 2
    $\begingroup$ In the first equality, $x$ is fixed by design therefore it must be independent of $\epsilon$ which implies that $E[\epsilon|x]=E[\epsilon]$. In the second equality $X$ is random hence we cannot assert that $E[\epsilon|X]=E[\epsilon]$ and because we cannot assert this the second equality does not need to hold (i.e. $E[Y|X]$ need not equal $E[Y]$). $\endgroup$ Commented Feb 20, 2022 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.