In simple regression model regressors are treated as fixed rather than stochastic. Whoever picks the experimental values for the regressors, decides in which frequency to include each value. This can be equally weighted (i.e 10 samples of 20mg medicine, 10 samples of 30mg medicine, 10 samples of 0mg-placebo) or in some other way.

Once the model is built and coefficients are determined, when it comes to use the model, it will be illogical to run the model with values of different frequency. It will be even more questionable to use values that did not exist in the experiment in the first place.

Moreover assuming that $Xs$ are fixed entails the following distributions:

$Y_i \sim N(\beta x_i,\sigma)$

Now if we consider the random sequence


It appears to me that it is not an exchangeable sequence without having a random $X$ to index them. Each $Y_i$ has a different mean. Hence the inference is not converging to something meaningful. $\hat\beta$ that is found is implicitly linked to the frequency of the values picked by the designer. Changing the frequencies changes the $\hat\beta$. The difference with respect to the random predictor case is that in the random case we acknowledge that there is a distribution, we don't get to mess with it during the estimation, and it will be maintained into the future for the sake of making predictions and having a comfort in terms of average prediction error. In the fixed case nothing can be said about the frequency of the future usage.

Another interesting aspect is that variability of $\hat\beta$ with respect to the design frequencies sneaks into cross-validation procedures. How can one safely split the design matrix into folds if we cannot assume convergence to the same $\hat\beta$ in the limit for each fold? We simply cannot sample from the design matrix if we consider it fixed. Same argument applies to various resampling procedures.

Moreover the fixed-X model doesn't actually contain a placeholder for future $x$ values, i.e. $Y_i \sim f_i(y_i)$. We are just lucky that fixed-X and random-X inference coincides under correct specification, so that we can abuse notation as if we have a placeholder.

For this reason I conclude, perhaps ignorantly, either there is no such thing as a fixed regressor (hence it is a scientific conspiracy), or there is a different way of looking at this to rationalize it.



On section 3, there is a discussion of the situation. It seems the problem emerges under misspecification. But this is almost always true when we have any data set at hand. Nobody knows the truth.

A point that has not received attention in the literature is that under general misspecification, the random versus fixed regressor distinction has implications for inference that do not vanish with the sample size.


One way to frame the question is in terms of different repeated sampling perspectives one can take. We can consider the distribution of the least squares estimator over repeated samples where we redraw the pairs Xi and Yi (the random regressor case), or we can consider the distribution over repeated samples where we keep the values of Xi fixed and only redraw the Yi (the fixed regressor case). Under general misspecification both the mean and variance of these two distributions will differ.

I would appreciate if somebody could make a connection to the exchangeability argument for an explanation, if there is a merit to it. How is it even working on correctly specified fixed-X situation, the basic proof is in every text book for $E[\hat\beta] = \beta$, but did we just get lucky?

  • 1
    $\begingroup$ What's "long run error rate" mean here exactly? $\endgroup$ Mar 2, 2015 at 12:44
  • $\begingroup$ Well you wouldn't expect performance to be the same on a very different set of predictor values. But you can calculate/simulate what it should be. E.g. the coefficient of determination obtained in a designed experiment is often greater than that obtained when the model fitted is applied to a sample from a naturally occurring population. $\endgroup$ Mar 2, 2015 at 12:56
  • 1
    $\begingroup$ I still cannot comprehend what you might mean by "long run error rate"--and understanding this seems to be essential to understanding the question. Calling it "a formula to accumulate" really doesn't tell us anything. Could you please edit the question to explain this key concept? $\endgroup$
    – whuber
    Mar 2, 2015 at 21:39
  • 1
    $\begingroup$ Please revise it then! $\endgroup$
    – whuber
    Mar 3, 2015 at 17:20
  • 3
    $\begingroup$ Can you show what you mean by changing coefficients when changing frequencies? I mean show it in an experimental setup, not in a mental experiment. Maybe you have the lab results that demonstrate it. It just sounds bizarre to me. I've done experimental physics long ago, and this claim doesn't correlate with my experience at all. $\endgroup$
    – Aksakal
    Oct 27, 2017 at 15:30

1 Answer 1

  1. A regression model gives predictions of the response conditional on predictor values; so there's no problem in applying a model fitted to one set of predictor values fixed by design to another set of predictor values, even if the latter are randomly sampled from a population. With an experimental design matrix $X$, the expectation & variance of the predicted response $\hat y$ for a (new) predictor vector $x$ are given by $$\operatorname{E}{\hat y \,|\, x} = x^\mathrm{T}\beta$$ $$\operatorname{Var}{\hat y\,|\,x}=\sigma^2\left(1+x^\mathrm{T}(X^\mathrm{T}X)^{-1}x\right)$$ where $\beta$ is the coefficient vector & $\sigma^2$ is the error variance—so the particular predictor values used for the fit don't affect the expectation of predictions, but do affect the variation in their precision throughout predictor space. Note that any aggregate fit metrics, say root mean square error of predictions, don't carry over from the experiment to the new sample.

  2. The above discussion assumes the model is right: in practice there will be extra-statistical considerations when applying it. You need to think about e.g. variation of effects that weren't investigated in the original experiment, the reliability of extrapolation into new regions of predictor space, selection bias in the population, & whether experimental manipulation is comparable to a natural cause. An engineer might model resistivity as a linear function of temperature from experimental data & be confident in applying the model to a particular collection of resistors used in a circuit board. The medical researcher in your example might assert that the medicine reduces blood cholesterol level, & confidently predict the results of further experiments; but would be unlikely to claim that, in a random sample from, say, all hospital admissions, those patients taking the medicine would have lower cholesterol levels than those who weren't.

  • $\begingroup$ Exploration doesn't have to be into the new regions. If model's repeated usage for values in the design matrix has a different frequency (but same values), then this is still not the correct model. Design matrix will have to be reweighted to accomodate. Basically this model is not good for anything. $\endgroup$ Mar 2, 2015 at 13:56
  • 1
    $\begingroup$ Don't want to play burden tennis, but can you give a hint as to why you think it's not? Or perhaps an example of what could go wrong? $\endgroup$ Mar 2, 2015 at 13:59
  • 4
    $\begingroup$ Note that we are not modeling the process that generates the design matrix, merely the relationship between the entries in the design matrix and the target variable. That relationship should not depend on how the design matrix was generated. If it does, there's something missing from the model, namely, the relationship between how the design matrix was generated and the target variable. Analogously, consider $y = 3x + \epsilon$; the "$3$" remains the same whether or not I generate $x$ from a binomial$(5,0.2)$ distribution or from a Gaussian$(0,1)$ distribution. $\endgroup$
    – jbowman
    Oct 30, 2017 at 15:49
  • 1
    $\begingroup$ @jbowman: Wish I'd put that so clearly. $\endgroup$ Oct 30, 2017 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.