11
$\begingroup$

When we calculate the standard error of a regression coefficient, we do not account for the randomness in the design matrix $X$. In OLS for instance, we calculate $\text{var}(\hat{\beta})$ as $\text{var}((X^TX)^{-1}X^TY) = \sigma^2(X^TX)^{-1}$

If the $X$ were considered random, the law of total variance would, in a sense, demand the additional contribution of the variance of $X$ as well. i.e.

$$\text{var}(\hat{\beta}) = \text{var}(E(\hat{\beta}|X)) + E(\text{var}(\hat{\beta}|X)).$$

Which, if the OLS estimator is truly unbiased, the first term vanishes since the expectation is a constant. The second term actually becomes: $\sigma^2 \text{cov}(X)^{-1}$.

  1. If a parametric model for $X$ is known, why don't we replace $X^TX$ with the actual covariance estimate. For instance, if $X$ is randomized treatment assignment, should the binomial variance $E(X)(1-E(X))$ be a more efficient estimate?

  2. Why don't we consider using flexible nonparametric models to estimate the possible sources of bias in in the OLS estimate and properly account for the sensitivity to design (i.e. the distribution of $X$) in the first law-of-total variance term $\text{var}(E(\hat{\beta}|X))$?

$\endgroup$
  • 2
    $\begingroup$ Why does a mathematical law "demand" anything? We use a model to reason with data to address particular objectives. When those are to understand or predict the conditional response based on an observed or measured value $X,$ the variation in $X$ would have little (if anything) at all to do with the substantive question--indeed, incorporating this variation in our procedures would seem to be altogether wrong, misleading, or even nonsensical. Answering your question therefore seems to comes down to assessing the frequencies with which different kinds of statistical problems are encountered. $\endgroup$ – whuber May 30 at 18:36
  • 1
    $\begingroup$ @whuber My focus is on inference. The law of total variance seems to be more inline with the frequentist interpretation of study results. We often talk of "if the study were replicated"... without accounting for the fact that the distribution of $X$ might differ if the study were replicated. The balance of sex could be 40% in one sample but 60% in another simply as a random consequence of how the study were obtained. Ironically, the bootstrap reflects this but does not generate any variability in outcome for a particular combination of covariates. $\endgroup$ – AdamO May 30 at 19:16
  • 2
    $\begingroup$ First off, many studies put $X$ under experimental control, so it's not even random. Second, observational studies (where $X$ is random) often are interested only in inference about the conditional distribution of $Y.$ Thus, focusing on inference does not distinguish one situation from the other. When the full (joint) distribution is of interest, you will see many people resorting to forms of correlation analysis or various multivariate procedures. There's no such thing as "the" bootstrap, because in this situation how you resample depends on your objectives as well as your model. $\endgroup$ – whuber May 30 at 20:14
  • 1
    $\begingroup$ @whuber Experimental control is randomly assigned at the point of study entry. As I mentioned, this is a compelling case: say the randomization is Bernoulli. Why use an empirical estimate of $\text{cov}(X) = X^TX$? Use maximum likelihood: $\text{cov}(X) = E(X)(1-E(X))$? You're correct about bootstrap, I was referring to non-parametric (unconditional) bootstrap where "rows" of data are sampled with replacement. $\endgroup$ – AdamO May 30 at 20:40
  • 2
    $\begingroup$ Outside of particular, anomalous cases, it doesn't really matter if $X_1$ is random, what matters is if there is measurement error in $X_1$. If so, OLS methods would lead to biased & lower powered estimates of $\beta_1$. In that case, errors in variables methods should be used. $\endgroup$ – gung Jun 12 at 16:52
7
$\begingroup$

Your question (plus further commentary in the comments) appears to be mostly interested in the case where we have a randomised controlled trial where the researcher randomly assigns one or more of the explanatory variables, based on some randomisation design. In this context, you want to know why we use a model that treats the explanatory variables as known constants, rather than treating them as random variables from the sampling distribution imposed by the randomisation. (Your question is broader than this, but this seems to be the case of primary interest in the commentary, so this is the one I will address.)

The reason that we condition on the explanatory variables, in this context, is that in a regression problem for an RCT, we are still interested in the conditional distribution of the response variable given the predictors. Indeed, in an RCT we are interested in determining the causal effects of an explanatory variable $X$ on the response variable $Y$, which we are going to determine via inference about the conditional distribution (subject to some protocols to prevent confounding). The randomisation is imposed to break dependence between the explanatory variable $X$ and any would-be confounding variables (i.e., prevent back-door associations).$^\dagger$ However, the object of inference in the problem is still the conditional distribution of the response variable given the explanatory variables. Thus, it still makes sense to estimate the parameters in this conditional distribution, using estimation methods that have good properties for inferring the conditional distribution.

That is the normal case that applies for an RCT using regression techniques. Of course, there are some situations where we have other interests, and we might indeed want to incorporate uncertainty about the explanatory variables. Incorporating uncertainty in the explanatory variables generally occurs in two cases:

  • (1) When we go beyond regression analysis and into multivariate analysis we are then interested is in the joint distribution of the explanatory and response variables, rather than just the conditional distribution of the latter given the former. There may be applications where this is our interest, and so we would then go beyond regression analysis, and incorporate information about the distribution of the explanatory variables.

  • (2) In some regression applications our interest is in the conditional distribution of the response variable conditional on an underlying unobserved explanatory variable, where we assume that the observed explanatory variables was subject to error ("errors-in-variables"). In this case we incorporate uncertainty via "errors-in-variables". The reason for this is that our interest in these cases is in the conditional distribution, conditional on an unobserved underlying variable.

Note that both of these cases are mathematically more complicated than regression analysis, so if we can get away with using regression analysis, that is generally preferable. In any case, in most applications of regression analysis, the goal is to make an inference about the conditional distribution of the response, given the observable explanatory variables, so these generalisations become unnecessary.


$^\dagger$ Note that randomisation severs causal effects from confounding variables to the randomised variable, but it does not sever causal effects from the randomised variable to the confounding variables, and then to the response. This means that other protocols (e.g., placebos, blinding, etc.) may be required to fully sever all back-door associations in a causal analysis.

$\endgroup$
  • 2
    $\begingroup$ Nice answer. I would add that AFAIK if you have gaussian errors-in-variables and gaussian error-in-response than the normal regression method works and it becomes an issue only if you a) observed response without error b) have a different response distribution $\endgroup$ – Martin Modrák May 31 at 13:25
1
$\begingroup$

The title "errors in variables" and the content of the question seems different, as it asks about why we do not take into account the variation in $X$ when modelling the conditional response, that is, in inference for regression parameters. Those two preoccupations seems orthogonal to me, so here I respond to the content.

I have answered to a similar question before, What is the difference between conditioning on regressors vs. treating them as fixed?, so here I will copy part of my answer there:

I will try to flesh out an argument for conditioning on regressors somewhat more formally. Let $(Y,X)$ be a random vector, and interest is in regression $Y$ on $X$, where regression is taken to mean the conditional expectation of $Y$ on $X$. Under multinormal assumptions that will be a linear function, but our arguments do not depend on that. We start with factoring the joint density in the usual way $$ f(y,x) = f(y\mid x) f(x) $$ but those functions are not known so we use a parameterized model $$ f(y,x; \theta, \psi)=f_\theta(y \mid x) f_\psi(x) $$ where $\theta$ parameterizes the conditional distribution and $\psi$ the marginal distribution of $X$. In the normal linear model we can have $\theta=(\beta, \sigma^2)$ but that is not assumed. The full parameter space of $(\theta,\psi)$ is $\Theta \times \Psi$, a Cartesian product, and the two parameters have no part in common.

This can be interpreted as a factorization of the statistical experiment, (or of the data generation process, DGP), first $X$ is generated according to $f_\psi(x)$, and as a second step, $Y$ is generated according to the conditional density $f_\theta(y \mid X=x)$. Note that the first step does not use any knowledge about $\theta$, that enters only in the second step. The statistic $X$ is ancillary for $\theta$, see https://en.wikipedia.org/wiki/Ancillary_statistic.

But, depending on the results of the first step, the second step could be more or less informative about $\theta$. If the distribution given by $f_\psi(x)$ have very low variance, say, the observed $x$'s will be concentrated in a small region, so it will be more difficult to estimate $\theta$. So, the first part of this two-step experiment determines the precision with which $\theta$ can be estimated. Therefore it is natural to condition on $X=x$ in inference about the regression parameters. That is the conditionality argument, and the outline above makes clear its assumptions.

In designed experiments its assumption will mostly hold, often with observational data not. Some examples of problems will be: regression with lagged responses as predictors. Conditioning on the predictors in this case will also condition on the response! (I will add more examples).

One book which discusses this problems in a lot of detail is Information and exponential families: In statistical theory by O. E Barndorff-Nielsen. See especially chapter 4. The author says the separation logic in this situation is however seldom explicated but gives the following references: R A Fisher (1956) Statistical Methods and Scientific Inference $\S 4.3$ and Sverdrup (1966) The present state of the decision theory and the Neyman-Pearson theory.


The factorization used here is somewhat similar in spirit to the factorization theorem of sufficient statistics. If focus is on the regression parameters $\theta$, and the distribution of $X$ do not depend on $\theta$, then how could the distribution of (or variation in) $X$ contain information about $\theta$?

This separation argument is helpful also because it points to the cases where it cannot be used, for instance regression with lagged responses as predictors.

$\endgroup$
  • $\begingroup$ I appreciate the question focused on OLS, but to make sure I've understood the implications of your answer I was wondering how would this play out in partial least squares regression? Since the data reduction of $X$ is partially dependent on $Y$ would this mean that $\theta$ and $\psi$ have common parameters? $\endgroup$ – ReneBt Jun 13 at 10:17
  • $\begingroup$ I don't know about PLS, but will try to think about it $\endgroup$ – kjetil b halvorsen Jun 13 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.