Fixed Regressor Conspiracy and Connection to Exchangeability

In simple regression model regressors are treated as fixed rather than stochastic. Whoever picks the experimental values for the regressors, decides in which frequency to include each value. This can be equally weighted (i.e 10 samples of 20mg medicine, 10 samples of 30mg medicine, 10 samples of 0mg-placebo) or in some other way.

Once the model is built and coefficients are determined, when it comes to use the model, it will be illogical to run the model with values of different frequency. It will be even more questionable to use values that did not exist in the experiment in the first place.

Moreover assuming that $$Xs$$ are fixed entails the following distributions:

$$Y_i \sim N(\beta x_i,\sigma)$$

Now if we consider the random sequence

$$Y_1,Y_2,...,Y_n$$

It appears to me that it is not an exchangeable sequence without having a random $$X$$ to index them. Each $$Y_i$$ has a different mean. Hence the inference is not converging to something meaningful. $$\hat\beta$$ that is found is implicitly linked to the frequency of the values picked by the designer. Changing the frequencies changes the $$\hat\beta$$. The difference with respect to the random predictor case is that in the random case we acknowledge that there is a distribution, we don't get to mess with it during the estimation, and it will be maintained into the future for the sake of making predictions and having a comfort in terms of average prediction error. In the fixed case nothing can be said about the frequency of the future usage.

Another interesting aspect is that variability of $$\hat\beta$$ with respect to the design frequencies sneaks into cross-validation procedures. How can one safely split the design matrix into folds if we cannot assume convergence to the same $$\hat\beta$$ in the limit for each fold? We simply cannot sample from the design matrix if we consider it fixed. Same argument applies to various resampling procedures.

Moreover the fixed-X model doesn't actually contain a placeholder for future $$x$$ values, i.e. $$Y_i \sim f_i(y_i)$$. We are just lucky that fixed-X and random-X inference coincides under correct specification, so that we can abuse notation as if we have a placeholder.

For this reason I conclude, perhaps ignorantly, either there is no such thing as a fixed regressor (hence it is a scientific conspiracy), or there is a different way of looking at this to rationalize it.

EDIT

https://economics.mit.edu/files/11856

On section 3, there is a discussion of the situation. It seems the problem emerges under misspecification. But this is almost always true when we have any data set at hand. Nobody knows the truth.

A point that has not received attention in the literature is that under general misspecification, the random versus fixed regressor distinction has implications for inference that do not vanish with the sample size.

..........

One way to frame the question is in terms of different repeated sampling perspectives one can take. We can consider the distribution of the least squares estimator over repeated samples where we redraw the pairs Xi and Yi (the random regressor case), or we can consider the distribution over repeated samples where we keep the values of Xi fixed and only redraw the Yi (the fixed regressor case). Under general misspecification both the mean and variance of these two distributions will differ.

I would appreciate if somebody could make a connection to the exchangeability argument for an explanation, if there is a merit to it. How is it even working on correctly specified fixed-X situation, the basic proof is in every text book for $$E[\hat\beta] = \beta$$, but did we just get lucky?

• What's "long run error rate" mean here exactly? Commented Mar 2, 2015 at 12:44
• Well you wouldn't expect performance to be the same on a very different set of predictor values. But you can calculate/simulate what it should be. E.g. the coefficient of determination obtained in a designed experiment is often greater than that obtained when the model fitted is applied to a sample from a naturally occurring population. Commented Mar 2, 2015 at 12:56
• I still cannot comprehend what you might mean by "long run error rate"--and understanding this seems to be essential to understanding the question. Calling it "a formula to accumulate" really doesn't tell us anything. Could you please edit the question to explain this key concept?
– whuber
Commented Mar 2, 2015 at 21:39
– whuber
Commented Mar 3, 2015 at 17:20
• Can you show what you mean by changing coefficients when changing frequencies? I mean show it in an experimental setup, not in a mental experiment. Maybe you have the lab results that demonstrate it. It just sounds bizarre to me. I've done experimental physics long ago, and this claim doesn't correlate with my experience at all. Commented Oct 27, 2017 at 15:30

1. A regression model gives predictions of the response conditional on predictor values; so there's no problem in applying a model fitted to one set of predictor values fixed by design to another set of predictor values, even if the latter are randomly sampled from a population. With an experimental design matrix $X$, the expectation & variance of the predicted response $\hat y$ for a (new) predictor vector $x$ are given by $$\operatorname{E}{\hat y \,|\, x} = x^\mathrm{T}\beta$$ $$\operatorname{Var}{\hat y\,|\,x}=\sigma^2\left(1+x^\mathrm{T}(X^\mathrm{T}X)^{-1}x\right)$$ where $\beta$ is the coefficient vector & $\sigma^2$ is the error variance—so the particular predictor values used for the fit don't affect the expectation of predictions, but do affect the variation in their precision throughout predictor space. Note that any aggregate fit metrics, say root mean square error of predictions, don't carry over from the experiment to the new sample.
• Note that we are not modeling the process that generates the design matrix, merely the relationship between the entries in the design matrix and the target variable. That relationship should not depend on how the design matrix was generated. If it does, there's something missing from the model, namely, the relationship between how the design matrix was generated and the target variable. Analogously, consider $y = 3x + \epsilon$; the "$3$" remains the same whether or not I generate $x$ from a binomial$(5,0.2)$ distribution or from a Gaussian$(0,1)$ distribution. Commented Oct 30, 2017 at 15:49