11
$\begingroup$

I know that "variable" means "values which vary." In a simple linear regression model :

$$Y=\beta_0+\beta_1X+\epsilon$$

$X$ is variable that is the values of $X$ vary. Why is $X$ not a random variable? What is the difference between a variable and a random variable?

$\endgroup$
7
  • 2
    $\begingroup$ X is a random variable in this example. You might instead be thinking of the distinction between fixed and random effects? These are terms used to describe the parameters being estimated. For example, $\beta_1$ could be thought of as a single value that is the same across observations, or as a random variable that can differ across observations. $\endgroup$
    – D L Dahly
    Commented Mar 2, 2015 at 15:04
  • $\begingroup$ @DLDahly, what about controlled experiments or dummy variables? Is $X$ a random variable even then? $\endgroup$ Commented Mar 2, 2015 at 15:05
  • 2
    $\begingroup$ Very roughly, random variable is a variable equipped with probability. In mathematics, you say $X$ is a variable if it is not fixed and can take multiple values. But does every possible value have the same chance (probability) of being selected? For example, if $X$ takes either 0 or 1 as its value, what can you say about the chance $X=1$ if $X$ is just a "variable"? If $X$ is a Bernoulli(p) "random variable", then you know $X=0$ with chance 1-p, and $X=1$ with chance p. $\endgroup$ Commented Mar 2, 2015 at 16:25
  • 2
    $\begingroup$ In regression, you can regard $X$ as a random variable if you know/assume its distribution, but it doesn't help because regression cares for only conditional distribution/expectation of $Y$ given $X$. That is, $X$ is fixed as a constant for the moment. $\endgroup$ Commented Mar 2, 2015 at 16:27
  • 1
    $\begingroup$ What a random variable is has been discussed extensively at stats.stackexchange.com/questions/50. $X$ might or might not be a random variable in this model, depending on how you view its values as arising. $\endgroup$
    – whuber
    Commented Mar 2, 2015 at 18:05

3 Answers 3

10
$\begingroup$

A variable is a symbol that represents some quantity. A variable is useful in mathematics because you can prove something without assuming the value of a variable and hence make a general statement over a range of values for that variable.

A random variable is a value that follows some probability distribution. In other words, it's a value that is subject to some randomness or chance.

In linear regression, $X$ may be viewed either as a random variable that is observed or it can be considered as a predetermined fixed value which, as LEP already discussed, the investigator chooses. As you've pointed out, we usually assume the later (whether or not this assumption is correct is another story). However, the OLS estimator is unbiased whether or not you treat $X$ as random and the estimate of the variance of the OLS estimator is unbiased for the variance of $\hat{\beta}_{OLS}$ whether or not you treat $X$ as random. These are a couple reasons people don't get too caught up in whether or not to assume $X$ is random in regression.

If you treat $X$ as random, I will show that the OLS estimator is still unbiased below.

Let $X$ be a random variable and let $\hat{\beta}_{OLS} = (X^{T}X)^{-1} X^{T} Y$.

$E(\hat{\beta}_{OLS})=E[E[\hat{\beta}_{OLS}|X]]=E[E[(X^{T}X)^{-1} X^{T} Y|X]]=E[(X^{T}X)^{-1} X^{T}E[ Y|X]] =E[(X^{T}X)^{-1} X^{T}X\beta] =E[\beta]=\beta$

If you treat $X$ as random, I will show that the estimate of the variance of $\hat{\beta}_{OLS}$ is unbiased for the unconditional variance below.

$Var(\hat{\beta}_{OLS})=Var(E(\hat{\beta}_{OLS}|X)) + E(Var(\hat{\beta}_{OLS}| X))=Var(\beta)+ E(Var(\hat{\beta}_{OLS}|X))=E(Var(\hat{\beta}_{OLS}|X))=E(\sigma^{2}( X^{T} X)^{-1})$

$\endgroup$
7
$\begingroup$

When you wrote down your equation, you did not list the assumptions: $$Y=\beta_0+\beta_1X+\epsilon$$

Why is X not a random variable?

Yes, it is often assumed (for simplicity of exposition in the intro statistics textbooks) that $X$ is fixed, or as you put it non-random.

It is fixed (non-random) in controlled experiments, i.e. mostly in natural sciences such as physics and biology. You can set the parameter $X$ at the level you're interested, and measure the response $Y$. In this case you make a set of assumptions such as Gauss-Markov theorem. For instance, feed the mice 1 mg of ascorbic acid and measure their hair loss. You control how much of the substance to administer.

However, it can be random, and it usually $is$ random in observational studies, i.e. 99% of all economics and social sciences alike. I can't set Dow-Jones Index (DJIA) at the arbitrary level, and measure the response in GDP (gross domestic product). I can only observe both, and whatever it is DJIA the day of my observation, that's my $X$. That's why the $X$ is random. In this case I have to use a different set of assumptions than the controlled experiment above. Imagine, now how difficult it is to establish causality between DJIA and DGP, unlike the case with mice when I decided how much of what to feed.

Here's additional reading:

  • The Gauss-Markov Theorem and Random Regressors. Author(s): Juliet Popper Shaffer. The American Statistician, Vol. 45, No. 4 (Nov., 1991), pp. 269-273
  • "Gauss–Markov Assumptions for Observational Research (Arbitrary x)" in Encyclopedia of Research Design, p.532:

A parallel but stricter set of Gauss–Markov assumptions is typically applied in practice in the case of observational data, where the researcher cannot assume that x is fixed in repeated samples.

$\endgroup$
0
$\begingroup$

Theoretically speaking the outcomes of an experiment (experiment = a random procedure) can be numerical (i.e rolling a die) or can be mapped to numbers by the designer (i.e flipping a coin, with outcomes 1=head and 0=tail).

This numerical representation of the outcomes defines the random variables.

In these examples, we can tell that there is some chance involved, in other words we don't control the experiment 100%. So a random variable is linked to observations in the real world, where uncertainty is involved, and that's where the "randomness" comes from.

Most importantly, as others have already pointed out, a random variable x (which is either discrete or continuous) is quantified by a probability density function (pdf). So we say that, random variables have distributions.

Now, in the case of linear regression, you ALREADY know the values of X and from that you try to figure out the values of Y, in other words, Y is the random variable (as you still don't have its values and they will depend on Xs' values).

Resources:

[1] Can someone help to explain the difference between independent and random?

[2] https://amsi.org.au/ESA_Senior_Years/SeniorTopic4/4_md/SeniorTopic4c.html

[3] Independent variable = Random variable?

$\endgroup$
1
  • 1
    $\begingroup$ Actually, if we're referring specifically to the OLS estimator, one of the assumptions is that the independent variables should be deterministic and not stochastic. $\endgroup$ Commented Feb 10, 2020 at 11:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.