20
$\begingroup$

I'm slightly confused if an independent variable (also called predictor or feature) in a statistical model, for example the $X$ in linear regression $Y=\beta_0+\beta_1 X$, is a random variable ?

$\endgroup$
  • 11
    $\begingroup$ The linear model is conditional on $X$, hence whether or not it is random should not matter. $\endgroup$ – Xi'an Nov 15 '16 at 12:25
  • 4
    $\begingroup$ Check this. Good question, BTW. $\endgroup$ – Antoni Parellada Nov 15 '16 at 13:28
  • $\begingroup$ @Xi'an, in the fixed design the linear model assumptions are not conditioned on $X$, see my answer. So, it does matter a lot. That is the reason why experiments are so much easier to interpret than observational study results $\endgroup$ – Aksakal Sep 19 '17 at 13:55
16
$\begingroup$

There are two common formulations of linear regression. To focus on the concepts, I will abstract them somewhat. The mathematical description is a little more involved than the English description, so let's begin with the latter:

Linear regression is a model in which a response $Y$ is assumed to be random with a distribution determined by regressors $X$ via a linear map $\beta(X)$ and, possibly, by other parameters $\theta$.

In most cases the set of possible distributions is a location family with parameters $\alpha$ and $\theta$ and $\beta(X)$ gives the parameter $\alpha$. The archetypical example is ordinary regression in which the set of distributions is the Normal family $\mathcal{N}(\mu, \sigma)$ and $\mu=\beta(X)$ is a linear function of the regressors.

Because I have not yet described this mathematically, it's still an open question what kinds of mathematical objects $X$, $Y$, $\beta$, and $\theta$ refer to--and I believe that is the main issue in this thread. Although one can make various (equivalent) choices, most will be equivalent to, or special cases, of the following description.


  1. Fixed regressors. The regressors are represented as real vectors $X\in\mathbb{R}^p$. The response is a random variable $Y:\Omega\to\mathbb{R}$ (where $\Omega$ is endowed with a sigma field and probability). The model is a function $f:\mathbb{R}\times\Theta\to M^d$ (or, if you like, a set of functions $\mathbb{R}\to M^d$ parameterized by $\Theta$). $M^d$ is a finite dimensional topological (usually second differentiable) submanifold (or submanifold-with-boundary) of dimension $d$ of the space of probability distributions. $f$ is usually taken to be continuous (or sufficiently differentiable). $\Theta\subset\mathbb{R}^{d-1}$ are the "nuisance parameters." It is supposed that the distribution of $Y$ is $f(\beta(X), \theta)$ for some unknown dual vector $\beta\in\mathbb{R}^{p*}$ (the "regression coefficients") and unknown $\theta\in\Theta$. We may write this $$Y \sim f(\beta(X), \theta).$$

  2. Random regressors. The regressors and response are a $p+1$ dimensional vector-valued random variable $Z = (X,Y): \Omega^\prime \to \mathbb{R}^p \times \mathbb{R}$. The model $f$ is the same kind of object as before, but now it gives the conditional probability $$ Y|X \sim f(\beta(X), \theta).$$

The mathematical description is useless without some prescription telling how it is intended to be applied to data. In the fixed regressor case we conceive of $X$ as being specified by the experimenter. Thus it might help to view $\Omega$ as a product $\mathbb{R}^p\times \Omega^\prime$ endowed with a product sigma algebra. The experimenter determines $X$ and nature determines (some unknown, abstract) $\omega\in\Omega^\prime$. In the random regressor case, nature determines $\omega\in\Omega^\prime$, the $X$-component of the random variable $\pi_X(Z(\omega))$ determines $X$ (which is "observed"), and we now have an ordered pair $(X(\omega), \omega)) \in \Omega$ exactly as in the fixed regressor case.


The archetypical example of multiple linear regression (which I will express using standard notation for the objects rather than this more general one) is that $$f(\beta(X), \sigma)=\mathcal{N}(\beta(x), \sigma)$$ for some constant $\sigma \in \Theta = \mathbb{R}^{+}$. As $x$ varies throughout $\mathbb{R}^p$, its image differentiably traces out a one-dimensional subset--a curve--in the two-dimensional manifold of Normal distributions.

When--in any fashion whatsoever--$\beta$ is estimated as $\hat\beta$ and $\sigma$ as $\hat\sigma$, the value of $\hat\beta(x)$ is the predicted value for $x$--whether $x$ is controlled by the experimenter (case 1) or is observed by him (case 2). If we either set a value (case 1) or observe a realization (case 2) $x$ of $X$, then the response $Y$ associated with that $X$ is a random variable whose distribution is $\mathcal{N}(\beta(x), \sigma)$, which is unknown but estimated to be $\mathcal{N}(\hat\beta(x), \hat\sigma)$.

$\endgroup$
  • $\begingroup$ Let me just mention, that this is a fantastic answer (but probably not for everybody). $\endgroup$ – l7ll7 Jan 14 '17 at 23:48
  • 1
    $\begingroup$ P.S. Do you know of any book, where these foundational question are explained as precisely as you did here ? As a mathematician, all the books I found reflected the other answers here, that are much less precise from a mathematical point of view. (This doesn't make them bad, of course, it's just that those books are not for me - I would love a book that is more precise, like this answer.) $\endgroup$ – l7ll7 Jan 15 '17 at 0:24
6
$\begingroup$

First of all, @whuber gave an excellent answer. I'll give it a different take, maybe simpler in some sense, also with a reference to a text.

MOTIVATION

$X$ can be random or fixed in the regression formulation. This depends on your problem. For so called observational studies it has to be random, and for experiments it usually is fixed.

Example one. I'm studying the impact of exposure to electron radiation on the hardness of a metal part. So, I take a few samples of the metal part and expose the to varying levels of radiation. My exposure level is X, and it's fixed, because I set to the levels that I chose. I fully control the conditions of the experiment, or at least try to. I can do the same with other parameters, such as temperature and humidity.

Example two. You're studying the impact of economy on frequency of occurrences of fraud in credit card applications. So, you regress the fraud event counts on GDP. You do not control GDP, you can't set to a desired level. Moreover, you probably want to look at multivariate regressions, so you have other variables such as unemployment, and now you have a combination of values in X, which you observe, but do not control. In this case X is random.

Example three. You are studying the efficacy of new pesticide in field, i.e. not in the lab conditions, but in the actual experimental farm. In this case you can control something, e.g. you can control the amount of pesticide to put. However, you do not control everything, e.g. weather or soil conditions. Ok, you can control the soil to some extent, but not completely. This is an in-between case, where some conditions are observed and some conditions are controlled. There's this entire field of study called experimental design that is really focused on this third case, where agriculture research is one of the biggest applications of it.

MATH

Here goes the mathematical part of an answer. There's a set of assumptions that are usually presented when studying linear regression, called Gauss-Markov conditions. They are very theoretical and nobody bothers to prove that they hold in any practical set up. However, they are very useful in understanding the limitations of ordinary least squares (OLS) method.

So, the set of assumptions is different for random and fixed X, which roughly correspond to observational vs. experimental studies. Roughly, because as I shown in the third example, sometimes we're really in-between the extremes. I found the "Gauss-Markov" theorem section in Encyclopedia of Research Design by Salkind is a good place to start, it's available in Google Books.

The differing assumptions of the fixed design are as follows for the usual regression model $Y=X\beta+\varepsilon$:

  • $E[\varepsilon]=0$
  • Homoscedasticity, $E[\varepsilon^2]=\sigma^2$
  • No serial correlation, $E[\varepsilon_i,\varepsilon_j]=0$

vs. the same assumptions in the random design:

  • $E[\varepsilon|X]=0$
  • Homoscedasticity, $E[\varepsilon^2|X]=\sigma^2$
  • No serial correlation, $E[\varepsilon_i,\varepsilon_j|X]=0$

As you can see the difference is in conditioning the assumptions on the design matrix for the random design. Conditioning makes these stronger assumptions. For instance, we are not just saying, like in fixed design, that the errors have zero mean; in random design we also say they're not dependent on X, covariates.

$\endgroup$
2
$\begingroup$

In statistics a random variable is quantity that varies randomly in some way. You can find a good discussion in this excellent CV thread: What is meant by a “random variable”?

In a regression model, the predictor variables (X-variables, explanatory variables, covariates, etc.) are assumed to be fixed and known. They are not assumed to be random. All of the randomness in the model is assumed to be in the error term. Consider a simple linear regression model as standardly formulated:
$$ Y = \beta_0 + \beta_1 X + \varepsilon \\ \text{where } \varepsilon\sim\mathcal N(0, \sigma^2) $$ The error term, $\varepsilon$, is a random variable and is the source of the randomness in the model. As a result of the error term, $Y$ is a random variable as well. But $X$ is not assumed to be a random variable. (Of course, it might be a random variable in reality, but that is not assumed or reflected in the model.)

$\endgroup$
  • $\begingroup$ So you mean $X$ is a constant ? Because that is the only other way to make sense of $X$ from a mathematical point of view, since $\varepsilon$ is a random variable and addition is only defined between two random variables and not "something else" + random variable. Though one of the two random variables could be constant, which is the case I'm referring to. $\endgroup$ – l7ll7 Nov 15 '16 at 23:14
  • $\begingroup$ P.S. I looked at all the explanations from said link and none very illuminating: Why ? Because none make the connection between random variables as probabilists understand it vs. how statisticians understand it. So some answers restate the standard, precise probability theory definition, while others restate the (yet unclear to me) vague statistical definition. But none really explain the connection between these two concepts.(The only exception is the long ticket-in-a-box model answer, which may show some promise, but even so [...] $\endgroup$ – l7ll7 Nov 15 '16 at 23:35
  • $\begingroup$ the difference wasn't fleshed out clearly enough to be strikingly illuminating; I'll have to meditate on this specific answer to see if there's any value to it) $\endgroup$ – l7ll7 Nov 15 '16 at 23:35
  • $\begingroup$ @user10324, if you like, you can think of $X$ as a set of constants. You could also think of it as a non-random variable. $\endgroup$ – gung Nov 16 '16 at 0:33
  • $\begingroup$ No, the non-random variable way of thinking about it does not work, for two reasons: One, as I argued in the comments above, there is no such thing as a "variable" in mathematics, and two, even if it were, then addition in that case is not defined, as I argued in the comments above. $\endgroup$ – l7ll7 Nov 16 '16 at 9:51
1
$\begingroup$

Not sure if I understand the question, but if you're just asking, "must an independent variable always be a random variable", then the answer is no.

An independent variable is a variable which is hypothesised to be correlated with the dependent variable. You then test whether this is the case through modelling (presumably regression analysis).

There are a lot of complications and "ifs, buts and maybes" here, so I would suggest getting a copy of a basic econometrics or statistics book covering regression analysis and reading it thoroughly, or else getting the class notes from a basic statistics/econometrics course online if possible.

$\endgroup$
  • $\begingroup$ Ok, but what is it, if it is not a random variable ? Just a (therefore deterministic) function ? I'm confused regarding the mathematical nature of the object "$X$". Actually, I found in the meantime a textbook, Probability and Statistics by Papoulis, where on page 149 he says "given two random variables $X$ and $Y$ [...]" and then goes on to explain how to regress $X$ on $Y$. So he seems to understand $X$ as a random variable ? $\endgroup$ – l7ll7 Nov 15 '16 at 14:25
  • $\begingroup$ P.S. I want to add that there is no such thing as a "variable" in mathematics when you look at it as a "standalone" objects (my background is maths). Variables in mathematics are just parts of standalone objects (e.g. arguments of function), but have no standalone meaning. If I would just write "x" in mathematics, it could mean the function $x\mapsto x$, or it could be a specific number, if $x$ was assigned a values previously, but we don't have just $x$. And since log. regression is a mathematical model, I'm interested in the mathematical meaning of $X$. $\endgroup$ – l7ll7 Nov 15 '16 at 14:37
  • $\begingroup$ It sounds as though you have a much greater understanding of maths than me. I'm just giving you the standard university undergraduate econometrics/statistics answer. I wonder if perhaps you might be overthinking it a bit, at least from the perspective of practical analysis. Regarding the quote from that book, my interpretation of that is that the specific x and y to which he is referring are random - but that doesn't mean that any x or any y are random. $\endgroup$ – Statsanalyst Nov 15 '16 at 16:03
  • $\begingroup$ e.g. the dependent variable in a model for voting trends in UK politics might be the number of votes received by the Conservative candidate in each constituency (Riding to Canadians, District to Americans), and the independent variable might be average house prices (a proxy for wealth/income in the UK). Neither of these is a "random" variable as I understand it, but this would be a perfectly reasonable thing to model. $\endgroup$ – Statsanalyst Nov 15 '16 at 16:10
  • $\begingroup$ Ok, that's is good to know what kind of answers I can expect/is the standard at econometrics/statistics departments and I appreciate that feedback very much (I would upvote again, but I can't since I already did). The problem with mathematics is "once you go black you never go back": Yearlong training in mathematical precision will induce a feeling of uneasiness if something is not crystal-clear fleshed out until one achieves claritiy [...] $\endgroup$ – l7ll7 Nov 15 '16 at 23:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.