Independent variable = Random variable?

Question

I'm slightly confused if an independent variable (also called predictor or feature) in a statistical model, for example the $X$ in linear regression $Y=\beta_0+\beta_1 X$, is a random variable ?

Answer 1

There are two common formulations of linear regression. To focus on the concepts, I will abstract them somewhat. The mathematical description is a little more involved than the English description, so let's begin with the latter:

Linear regression is a model in which a response $Y$ is assumed to be random with a distribution determined by regressors $X$ via a linear map $\beta(X)$ and, possibly, by other parameters $\theta$.

In most cases the set of possible distributions is a location family with parameters $\alpha$ and $\theta$ and $\beta(X)$ gives the parameter $\alpha$. The archetypical example is ordinary regression in which the set of distributions is the Normal family $\mathcal{N}(\mu, \sigma)$ and $\mu=\beta(X)$ is a linear function of the regressors.

Because I have not yet described this mathematically, it's still an open question what kinds of mathematical objects $X$, $Y$, $\beta$, and $\theta$ refer to--and I believe that is the main issue in this thread. Although one can make various (equivalent) choices, most will be equivalent to, or special cases, of the following description.

---

1. Fixed regressors. The regressors are represented as real vectors $X\in\mathbb{R}^p$. The response is a random variable $Y:\Omega\to\mathbb{R}$ (where $\Omega$ is endowed with a sigma field and probability). The model is a function $f:\mathbb{R}\times\Theta\to M^d$ (or, if you like, a set of functions $\mathbb{R}\to M^d$ parameterized by $\Theta$). $M^d$ is a finite dimensional topological (usually second differentiable) submanifold (or submanifold-with-boundary) of dimension $d$ of the space of probability distributions. $f$ is usually taken to be continuous (or sufficiently differentiable). $\Theta\subset\mathbb{R}^{d-1}$ are the "nuisance parameters." It is supposed that the distribution of $Y$ is $f(\beta(X), \theta)$ for some unknown dual vector $\beta\in\mathbb{R}^{p*}$ (the "regression coefficients") and unknown $\theta\in\Theta$. We may write this $$Y \sim f(\beta(X), \theta).$$

2. Random regressors. The regressors and response are a $p+1$ dimensional vector-valued random variable $Z = (X,Y): \Omega^\prime \to \mathbb{R}^p \times \mathbb{R}$. The model $f$ is the same kind of object as before, but now it gives the conditional probability $$ Y|X \sim f(\beta(X), \theta).$$

The mathematical description is useless without some prescription telling how it is intended to be applied to data. In the fixed regressor case we conceive of $X$ as being specified by the experimenter. Thus it might help to view $\Omega$ as a product $\mathbb{R}^p\times \Omega^\prime$ endowed with a product sigma algebra. The experimenter determines $X$ and nature determines (some unknown, abstract) $\omega\in\Omega^\prime$. In the random regressor case, nature determines $\omega\in\Omega^\prime$, the $X$-component of the random variable $\pi_X(Z(\omega))$ determines $X$ (which is "observed"), and we now have an ordered pair $(X(\omega), \omega)) \in \Omega$ exactly as in the fixed regressor case.

---

The archetypical example of multiple linear regression (which I will express using standard notation for the objects rather than this more general one) is that $$f(\beta(X), \sigma)=\mathcal{N}(\beta(x), \sigma)$$ for some constant $\sigma \in \Theta = \mathbb{R}^{+}$. As $x$ varies throughout $\mathbb{R}^p$, its image differentiably traces out a one-dimensional subset--a curve--in the two-dimensional manifold of Normal distributions.

When--in any fashion whatsoever--$\beta$ is estimated as $\hat\beta$ and $\sigma$ as $\hat\sigma$, the value of $\hat\beta(x)$ is the predicted value of $Y$ associated with $x$--whether $x$ is controlled by the experimenter (case 1) or is only observed (case 2). If we either set a value (case 1) or observe a realization (case 2) $x$ of $X$, then the response $Y$ associated with that $X$ is a random variable whose distribution is $\mathcal{N}(\beta(x), \sigma)$, which is unknown but estimated to be $\mathcal{N}(\hat\beta(x), \hat\sigma)$.

Answer 2

First of all, @whuber gave an excellent answer. I'll give it a different take, maybe simpler in some sense, also with a reference to a text.

MOTIVATION

$X$ can be random or fixed in the regression formulation. This depends on your problem. For so called observational studies it has to be random, and for experiments it usually is fixed.

Example one. I'm studying the impact of exposure to electron radiation on the hardness of a metal part. So, I take a few samples of the metal part and expose the to varying levels of radiation. My exposure level is X, and it's fixed, because I set to the levels that I chose. I fully control the conditions of the experiment, or at least try to. I can do the same with other parameters, such as temperature and humidity.

Example two. You're studying the impact of economy on frequency of occurrences of fraud in credit card applications. So, you regress the fraud event counts on GDP. You do not control GDP, you can't set to a desired level. Moreover, you probably want to look at multivariate regressions, so you have other variables such as unemployment, and now you have a combination of values in X, which you observe, but do not control. In this case X is random.

Example three. You are studying the efficacy of new pesticide in field, i.e. not in the lab conditions, but in the actual experimental farm. In this case you can control something, e.g. you can control the amount of pesticide to put. However, you do not control everything, e.g. weather or soil conditions. Ok, you can control the soil to some extent, but not completely. This is an in-between case, where some conditions are observed and some conditions are controlled. There's this entire field of study called experimental design that is really focused on this third case, where agriculture research is one of the biggest applications of it.

MATH

Here goes the mathematical part of an answer. There's a set of assumptions that are usually presented when studying linear regression, called Gauss-Markov conditions. They are very theoretical and nobody bothers to prove that they hold in any practical set up. However, they are very useful in understanding the limitations of ordinary least squares (OLS) method.

So, the set of assumptions is different for random and fixed X, which roughly correspond to observational vs. experimental studies. Roughly, because as I shown in the third example, sometimes we're really in-between the extremes. I found the "Gauss-Markov" theorem section in Encyclopedia of Research Design by Salkind is a good place to start, it's available in Google Books.

The differing assumptions of the fixed design are as follows for the usual regression model $Y=X\beta+\varepsilon$:

• $E[\varepsilon]=0$
• Homoscedasticity, $E[\varepsilon^2]=\sigma^2$
• No serial correlation, $E[\varepsilon_i,\varepsilon_j]=0$

vs. the same assumptions in the random design:

• $E[\varepsilon|X]=0$
• Homoscedasticity, $E[\varepsilon^2|X]=\sigma^2$
• No serial correlation, $E[\varepsilon_i,\varepsilon_j|X]=0$

As you can see the difference is in conditioning the assumptions on the design matrix for the random design. Conditioning makes these stronger assumptions. For instance, we are not just saying, like in fixed design, that the errors have zero mean; in random design we also say they're not dependent on X, covariates.

Answer 3

In statistics a random variable is quantity that varies randomly in some way. You can find a good discussion in this excellent CV thread: What is meant by a “random variable”?

In a regression model, the predictor variables (X-variables, explanatory variables, covariates, etc.) are assumed to be fixed and known. They are not assumed to be random. All of the randomness in the model is assumed to be in the error term. Consider a simple linear regression model as standardly formulated:
$$ Y = \beta_0 + \beta_1 X + \varepsilon \\ \text{where } \varepsilon\sim\mathcal N(0, \sigma^2) $$ The error term, $\varepsilon$, is a random variable and is the source of the randomness in the model. As a result of the error term, $Y$ is a random variable as well. But $X$ is not assumed to be a random variable. (Of course, it might be a random variable in reality, but that is not assumed or reflected in the model.)

Answer 4

Not sure if I understand the question, but if you're just asking, "must an independent variable always be a random variable", then the answer is no.

An independent variable is a variable which is hypothesised to be correlated with the dependent variable. You then test whether this is the case through modelling (presumably regression analysis).

There are a lot of complications and "ifs, buts and maybes" here, so I would suggest getting a copy of a basic econometrics or statistics book covering regression analysis and reading it thoroughly, or else getting the class notes from a basic statistics/econometrics course online if possible.