6
$\begingroup$

If granted omniscience and we know that $\beta_1$ in a multiple linear regression model is truly 0, what does that mean in words (and math notation)?

The model is: $Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + \epsilon$

Does $\beta_1 = 0$ mean:

  1. Empirically: "If we make infinite, perfect measurements of our data, $\hat\beta_1$ approaches 0 at the limit?" [Math: ??]
  2. Formally: "In the true, finite population of our data, $X_1$ has 0 association with $Y$ and their vectors of values are perfectly linearly independent, uncorrelated, and orthogonal?" [Math: ??]

What does $\beta_1 = 0$ mean, precisely?

$\endgroup$
  • 4
    $\begingroup$ Many of your separate questions come across as minor variations on each other. Also, if you ask short questions, much of the answer may turn out to be of the form: if you mean (a), then (A); if (b), then (B); and so forth. This strikes me as an interesting question, but answers could range back and forth across large swathes of statistical thinking, so despite the specificity here I am almost tempted to vote to close as too broad. I won't do so, but that is one signal to you, for what it's worth. $\endgroup$ – Nick Cox Mar 13 '15 at 13:49
  • $\begingroup$ @NickCox: I do thank you for your comments and signals. My apologies if my (evolutionarily pruning) questions are bad form as I try to precisely pin down a conceptual difficulty in canonical form. Still learning CV community ethics, assuming you signal against minor variations. I have tried to make question better but as I do not know the menu of (a), (b),.. I may have missed mark. In any event, thanks for guidance! $\endgroup$ – jtd Mar 13 '15 at 14:16
  • 2
    $\begingroup$ Can you describe the setting in a more precise manner? Given that you insist there is no error term in the model, it's not a "standard" regression model. What is $x_1,x_2,\beta_1,\beta_2$? Normally the $\beta_i$ are considered constants, so of course it cannot be that $\beta_1=0$ and $\beta_1\neq 0$. You must have something else in mind. $\endgroup$ – ekvall Mar 13 '15 at 18:27
  • $\begingroup$ @Student001: Sorry, my thinking was: "true value y = true intercept + 0*$x_1$ + true $\beta_2x_2$ + true error of zero." E.g., 4 = -1 + (0*3) + (0.5*10) + 0. It is not parsimonious but I think still valid. The and was because I incorrectly thought NickCox suggested I needed to limit question to 1 paradigm, 1 method and I worried that truth value of $\beta_1$ might change--even though that seems nonsense, I was out of my depth trying to be precise. That is now removed. $\endgroup$ – jtd Mar 13 '15 at 19:15
6
$\begingroup$

Part of the problem here is that your model is conceptually confused, as @CagdasOzgenc has correctly pointed out.(Edit: This has now been fixed.) I think I can address two of your specific questions.

  1. If $\beta_1 = 0$ (note the absence of the 'hat'), and standard assumptions hold, then empirically, as we make ever more measurements of our data, $\hat\beta_1$ will approach $0$ at the limit.

    In mathematical notation:
    $$ \lim_{N\to\infty} \hat\beta_1 = 0 $$

  2. If $\beta_1 = 0$, and $X_1$ and $X_2$ are uncorrelated, then formally, in the population of our data, $x_1$ is uncorrelated with $y$, linearly independent of $y$, but not necessarily independent of $y$*. (Edit: as @CagdasOzgenc points out, if we take the model provided as literally the DGP, independence holds as well.)

    In mathematical notation:
    $$ {\rm Cor}(x_1, y) = 0 $$

    * From the Wikipedia page on correlation and dependence, consider this figure:

    enter image description here

    The patterns in the bottom row all have correlation $0$, but show various patterns of dependence.

I don't understand the part after the bolded text.(Edit: The referenced portion of the question has now been removed.)

$\endgroup$
  • $\begingroup$ This is fantastic! Should we say, "$x_1$ is uncorrelated with $y$, linearly independent of $y$, but not necessarily independent of $y$.*" (psych.umn.edu/faculty/waller/classes/FA2010/Readings/…) $\endgroup$ – jtd Mar 13 '15 at 18:58
  • $\begingroup$ @jtd, yes, I think so. $\endgroup$ – gung - Reinstate Monica Mar 13 '15 at 19:04
  • 2
    $\begingroup$ Actually if $\beta_1$ = 0 and $y$ is generated by the model in question, it would also imply independence, as $x_1$ didn't take part. But this of course implies a directionality in data generation. If no directionality is assumed the uncorrelatedness can also not be concluded as all the dynamics may be captured by $\beta_2$. I need to ponder. $\endgroup$ – Cagdas Ozgenc Mar 13 '15 at 19:33
  • $\begingroup$ @CagdasOzgenc, good point. If we take the given model literally as the DGP, there is independence as well. I guess I was thinking of it as illustrative. I'll edit my answer. Let me know if you think it needs more. $\endgroup$ – gung - Reinstate Monica Mar 13 '15 at 19:36
  • $\begingroup$ @gung: If $\beta_1 = 0$ means $Cor(x_1,y) = 0$, does this mean simple $Cor(x_1,y)$ can help us determine if true $\beta_1 = 0$, but once we think $\beta_1 \ne 0$, then we must use linear regression to get more information? So linear regression does two steps $Cor()$ plus something. Also, should we actually say that $\beta_1 = 0$ means $Cor(x_1,y)|x_2???$ or is it a straight bivariate comparison? $\endgroup$ – jtd Mar 13 '15 at 19:44
5
$\begingroup$

You are confusing the real data generation process with a model trying to make an approximation of this process (this is quite normal, I also banged my head to the wall several times regarding this matter).

First of all

$y = \beta_0 + \beta_1x_1 + \beta_2x_2$

is a deterministic model. Usually in regression, the model is probabilistic hence, the correct representation of the model is

$y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \epsilon$ and $\epsilon$ ~ some distribution(usually normal)

Now suppose that above model is exactly capturing the underlying real data generation. In that case $\beta_1$ = 0 means it is always 0 in reality.

However when you propose the above model you need to estimate the parameters from sample data.

In that case

$\hat{\beta_1}$ (estimation of $\beta_1$) can be anything. The probability that it will be exactly 0 is 0. If in reality it is 0, and if you did proper sampling and used a consistent estimation procedure as sample size increases it will approach 0.

$\endgroup$
  • $\begingroup$ Thanks, perhaps I'm confusing things but I corrected $\beta_1 \to \hat\beta_1$ in #1. I'm talking about the real data generation process, not a model: we are omniscient, there are no errors, no samples, and the Truth is $\beta_1 = 0$, but what does $\beta_1 = 0$ mean in precise, unique language (and math) in this linear regression context? Your last sentence seems to agree with my first bullet point, "If in reality it is 0,..", is this your answer? $\endgroup$ – jtd Mar 13 '15 at 15:29
  • 1
    $\begingroup$ "You are confusing the real data generation process with a model trying to make an approximation of this process." This problem is endemic in undergrad econometrics. Then the problem is magnified by sloppy notation ("is $x$ a random variable or the realization of a random variable?") and over-emphasizing the "$Y$ as a function of $X$, plus some error" interpretation of regression over the "$E(Y|X)$" interpretation. $\endgroup$ – shadowtalker Mar 13 '15 at 22:33
4
$\begingroup$

Gung's answer is excellent, but I want to add an interpretation that I think goes underappreciated.

You wrote out the model as $$ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \varepsilon $$

You didn't specify this, but presumably $\varepsilon$ is an error term with $\operatorname{\mathbb{E}}\left(\varepsilon\,|\,X\right) = 0$ and $\varepsilon \perp X$. Here you are postulating a particular data-generating process: given $X_1$ and $X_2$, $Y$ is a deterministic function of $X_1$ and $X_2$, plus a random error term. This is what is usually taught in undergrad econometrics class.

Now just for the heck of it, define a function $\mu(x_1, x_2) = \beta_0 + \beta_1 x_1 + \beta_2 x_2$ so that the model can be written as $$ Y = \mu(X_1, X_2) + \varepsilon $$ or, perhaps more precisely, $$ Y\,|\,(X_1 = x_1, X_2 = x_2) = \mu(x_1, x_2) + \varepsilon $$

Remember that we assumed $\operatorname{\mathbb{E}}\left(\varepsilon\,|\,X_1 = x_1, X_2 = x_2\right) = 0$. Taking the hint, let's compute $$ \begin{align} \operatorname{\mathbb{E}}(Y\,|\,X_1 = x_1, X_2 = x_2) &= \operatorname{\mathbb{E}}(\mu(x_1, x_2)) &+& \operatorname{\mathbb{E}}\left(\varepsilon\,|\,X_1 = x_1, X_2 = x_2\right) \\ \operatorname{\mathbb{E}}(Y\,|\,X_1 = x_1, X_2 = x_2) &= \mu(x_1, x_2) &+& 0 \\ \operatorname{\mathbb{E}}(Y\,|\,X_1 = x_1, X_2 = x_2) &= \beta_0 + \beta_1 x_1 + \beta_2 x_2 \end{align} $$

This is powerful stuff: "regression line" is really the expected $Y$ as a function of $X$.

You asked what it means if $\beta_1 = 0$. In this interpretation, it means that the expectation of $Y$ does not depend on $X_1$. That is, $$\begin{align} \operatorname{\mathbb{E}}(Y\,|\,X_1 = x_1, X_2 = x_2) &= \beta_0 &+ 0 \cdot x_1 &+ \beta_2 x_2 \\ \operatorname{\mathbb{E}}(Y\,|\,X_1 = x_1, X_2 = x_2) &= \beta_0 &&+ \beta_2 x_2 \end{align}$$

In other words, $\beta_1 = 0$ means $X_1$ does not belong in the model. The slope of the regression line (i.e. the "conditional expectation line") is 0 with respect to $X_1$. Compare: $z = 2x + 2y$ and $z = 0x + 2y$

Now remember our second assumption that $\varepsilon \perp X$. From this we can conclude that in fact $Y \perp X_1$. We have already established that changing the value of $X_1$ has no effect on $\mu$, the average $Y$ given $X_1$ and $X_2$, but if $\varepsilon \perp X_1$ as well, then there's just nowhere else for $X_1$ to enter the data generating process. It doesn't affect the average $Y$ and it doesn't effect the variation of $Y$ around its average, so it just doesn't affect $Y$ at all.

Empirically, this means that any value we estimate for $\beta_1$, which we usually denote $\hat \beta_1$, should be close to zero. If we use OLS to fit the model, we know that $\operatorname{\mathbb{E}}(\hat \beta_1) = \beta_1 = 0$ and $\hat \beta_1 \xrightarrow[]{n \to \infty} \beta_1$. So the expectation of $\hat \beta_1$ will be zero, and $\hat \beta_1$ will approach zero as the sample grows.

$\endgroup$
  • $\begingroup$ +1. Note that you have one unfinished sentence ("In a 3D space..."). $\endgroup$ – amoeba says Reinstate Monica Mar 13 '15 at 22:39
  • $\begingroup$ @amoeba you're too quick for me. $\endgroup$ – shadowtalker Mar 13 '15 at 22:39
  • $\begingroup$ Unfortunately equality notation doesn't involve any direction. It is not possible to conclude that first we acquire the $Xs$ and then add some noise on top and generate $Y$. You may simply move $X_1$ to the left and $Y$ to the right. A simple example is $Weight = \beta Height + \epsilon $. If we sample "people" from a population of people, DGP view dissolves. For this reason at this point we cannot conclude more than the fact that every equation will be just a projection of reality. $Y = \beta_1 X_1 + \epsilon_1$ and $Y = \beta_2 X_2 + \epsilon_2$ both can be valid. $\endgroup$ – Cagdas Ozgenc Mar 14 '15 at 8:37
  • 1
    $\begingroup$ @CagdasOzgenc very good point. that's why I added all that ridiculous conditioning notation and the swapping of lower and upper case, but I could have been more explicit. $\endgroup$ – shadowtalker Mar 14 '15 at 10:42
  • 1
    $\begingroup$ And the DGP still can hold in that case. Height is determined somehow, then weight is drawn randomly afterwards. Even if it doesn't make biological sense (although it happens to be plausible here), it's an intuitive way to condition one variable on another. All models are false, etc $\endgroup$ – shadowtalker Mar 14 '15 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.