9
$\begingroup$

Recall the model for simple linear regression $$ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i. $$

I am reading up on the standard error of the coefficients $\beta_0$ and $\beta_1$. As an experiment I generated some linear data using $\beta_0 = 1$ and $\beta_1 = 2$ and added some Gaussian noise with unit variance. So then when I fit the data the lm function and used the summary function to examine the model I have the following output: \begin{align} \hat \beta_0 & = 1.21054 \quad \text{with Std. Error} = 0.11508, \\ \hat \beta_1 & = 1.87723 \quad \text{with Std. Error} = 0.09844. \end{align}

So how do I interpret the standard error values? For instance, take $\hat \beta_0$, precisely what is $0.11508$ telling me?

Obviously if I ran the simulation a second time, this time adding Gaussian noise with a higher amount of variance, the standard error would increase as the extra variance in the noise shows up as an increase in the standard error of the coefficients. But, if we consider the first simulation in isolation, then what does this value of $0.11508$ mean?

$\endgroup$
11
$\begingroup$

The standard error is the square root of an estimate of the sampling variability of $\hat\beta_j$ as an estimator of $\beta_j$, or $\sqrt{\widehat{Var}(\hat\beta_j)}$.

As this is many things in one sentence, step-by-step:

  1. "Square-root": should be self-explanatory, to turn a variance into a standard deviation (that turns out to be what we need in, for example, t-statistics and confidence intervals).
  2. "$\hat\beta_j$ as an estimator of $\beta_j$": we use the LS estimator to estimate the unknown parameter $\beta_j$.
  3. To do so, we make use of a sample from the underlying population. Had we drawn another sample (or were to draw a fresh one tomorrow, etc.) we would get another estimate $\hat\beta_j$. This is the source of sampling variability. We may summarize that variability through the variance, $Var(\hat\beta_j)$. An expression for this variance may be found, e.g., here.
  4. "An estimate of the sampling variability": $Var(\hat\beta_j)$ depends on unknown quantities (like the variance of the Gaussian noise that you generated), which must therefore be estimated, as captured by the formula $\widehat{Var}(\hat\beta_j)$. A formula for this estimator is, for example, given here, or, more introductory, here.
$\endgroup$
6
  • $\begingroup$ My question is about what this particular value of $0.11508$ means. I know we will get different outcomes if we draw other samples. But for a single sample in isolation: what does it tell me when the coefficient is $1.21054$ with a standard error of $0.11508$? $\endgroup$ – ManUtdBloke Sep 2 '20 at 12:32
  • 3
    $\begingroup$ In and of itself, it does not tell us too much, but it becomes interesting when coupled with the distribution of the LS estimator, which is normal under suitable assumptions. Since 95% of the realizations of a normal random variables are (roughly) fall into two standard deviations around the mean and since LS is unbiased (again, under suitable assumptions), we may, e.g., use the point estimate and the standard error to obtain a confidence interval, which is heuristically often interpreted as a range of plausible estimates for the true parameter. $\endgroup$ – Christoph Hanck Sep 2 '20 at 12:35
  • 1
  • 3
    $\begingroup$ Alternatively, point estimate/standard error is the t-statistic for the null hypothesis that the true coefficient is zero. So with your numbers, the t-statistic would far exceed two, the critical value at the 5% level, so that you would reject the null at that level. $\endgroup$ – Christoph Hanck Sep 2 '20 at 12:39
  • 1
    $\begingroup$ No, (e.g.) the first link in my comment above recalls how to build a confidence interval. What I want to alert to is that the language used there might indicate that confidence intervals are interpreted that is not afforded by their definition. $\endgroup$ – Christoph Hanck Sep 7 '20 at 8:05
4
$\begingroup$

It’s the usual definition of the standard error: the (estimated) standard deviation of the sampling distribution of $\hat{\beta}_0$.

If you were to replicate the work many times with new observations, you would get a distribution of values. Sometimes it would be higher than you observed this time, sometimes lower.

We use the standard error in parameter inference. Being loose, if the p-value on the parameter is less than $0.05$, corresponding to a point estimate about $2$ standard errors above or below $0$, then we might say that the population parameter is not zero, so that variable has a measurable impact on the outcome.

(There are all sorts of caveats about p-values, and discussing them really warrants a separate question (or a master’s degree in statistics).)

$\endgroup$
2
  • $\begingroup$ My question is about what this particular value of $0.11508$ means. I know we will get different outcomes if we draw other samples. But for a single sample in isolation: what does it tell me when the coefficient is $1.21054$ with a standard error of $0.11508$? $\endgroup$ – ManUtdBloke Sep 2 '20 at 12:32
  • $\begingroup$ It has to do with hypothesis testing and creating confidence intervals, same as usual. $\endgroup$ – Dave Sep 2 '20 at 13:24
2
$\begingroup$

If

\begin{align} \hat \beta_0 & = 1.21054 \quad \text{with Std. Error} = 0.11508, \\ \hat \beta_1 & = 1.87723 \quad \text{with Std. Error} = 0.09844. \end{align}

it means that the range of values for the coefficient estimates are

$$\hat \beta_0 = 1.21054 \pm 0.11508$$

and

$$\hat \beta_1 = 1.87723 \pm 0.09844$$

In other words, you can be confident that $\beta_0$ can take on values between $1.09546$ and $1.32562$.

As for your repeated question of

But for a single sample in isolation: what does it tell me when the coefficient is 1.21054 with a standard error of 0.11508

it is not relevant because the $\beta$s (and therefore the $\sigma(\beta)$s) are computed based on the entire sample set, not from one specific observation. A $\beta$ relates the entire $y$ sample set with the entire sample set being input for (one of) the corresponding $x$ vector.

I think what you mean to ask is, what does $\hat \beta_0 = 1.21054 \pm 0.11508$ mean for the output $\hat{y}$ of my fitted model if a new sample $x_i=0.2$ is observed. Well, since $y = \beta_0 x + \beta_0 x + \epsilon$, then the predicted output given that new input is

$$\hat{y}_i = (1.21054 \pm 0.11508) \times 0.2 + (\beta_1 \pm \sigma(\beta_1))\times 0.2 + \epsilon$$

$\endgroup$
4
  • $\begingroup$ I don't understand $$\hat \beta_0 = 1.21054 \pm 0.11508$$ $\hat \beta_0 = 1.21054$... I think I know what you're trying to say, but the notation here doesn't really make sense. $\endgroup$ – Do not reinstate Monica Sep 2 '20 at 14:48
  • $\begingroup$ In other words, you can be confident that $\beta_0$ can take on values between $1.09546$ and $1.32562$.--- What I am really interested in is precisely how confident can I be that $\beta_0$ will be within that range? $\endgroup$ – ManUtdBloke Sep 2 '20 at 15:45
  • $\begingroup$ Thsi answer and your last question seem to have a somewhat Bayesian flavor. The standard error cannot be used to make such confidence statements. It would be too broad to explain the Bayesian idea here, though, but there are many excellent posts on this site on this approach. $\endgroup$ – Christoph Hanck Sep 3 '20 at 4:26
  • $\begingroup$ @ChristophHanck The answer of Emil Bode below indicates that we can give confidence intervals based on these standard errors. Is this user incorrect? $\endgroup$ – ManUtdBloke Sep 6 '20 at 18:04
1
$\begingroup$

What you created here is a model, that tries to reflect reality. But of course, unless we are exceptionally lucky, the model will never reflect reality perfectly. And the standard deviations reflect how confident the model is about itself.

In your question, you stated you generated data with $\beta_0 = 1$ and $\beta_1=2$. Those numbers are the reality your model tries to reflect. Now suppose you didn't tell us those values, just your model. What can we say about your input?

The model tells us the most likely values are $\beta_0 = 1.21042$ and $\beta_1=1.87223$. But could it be that you that the actual values you put in (the reality) were $1.2$ and $1.9$? Well, therefore we have to look at the standard deviation.

With the given standard deviations, the model tells you it's $68 \%$ sure the true value of $\beta_0$ is in the range $1.09546 - 1.32562$ (minus 1 sd and plus 1 sd). And it's $95 \%$ sure the true value is in the range $0.98038 - 1.4407$ (2 sd away). For $\beta_1$, we can do a similar calculation. That means the numbers $1.2$ and $1.9$ are very reasonable guesses, but that $1$ and $2$ are also not too outlandish.

Now in reality, we often don't have access to the true values of $\beta_0$ and $\beta_1$. We can just take measurements, and make the best model we have. Or sometimes, theorists will come up with a theory that has to be tested on reality, to check whether the model is right or wrong.

As an experimental physicist, you'll run some experiments and maybe get the same values you got. You'll make a model, and can publish this to show that a theory that predicts $\beta_0 = 0$ and $\beta_1=5$ is most definitely wrong (if you can prove your experimental setup is correct). The values you got of $1.21$ and $1.87$ are basically your best guesses as to what the true values could be. But a theory that predicts $\beta_0=1$ and $\beta_1=2$ may well be correct.

Until you come up with an experiment that's more sensitive. Suppose you do the same, and get a model that shows: \begin{align} \hat \beta_0 & = 1.19554 \quad \text{with Std. Error} = 0.01279, \\ \hat \beta_1 & = 1.88341 \quad \text{with Std. Error} = 0.02369. \end{align}

These values allign quite well with your earlier result (showing there was likely no systemic error in your first experiment). But they have much narrower standard deviations, and now also show the theory with $\beta_0=1$ and $\beta_1=2$ is also wrong. But the guesses of $1.2$ and $1.9$ are still holding.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.