Meaning of standard error of the coefficients in a regression model?

Question

Recall the model for simple linear regression $$ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i. $$

I am reading up on the standard error of the coefficients $\beta_0$ and $\beta_1$. As an experiment I generated some linear data using $\beta_0 = 1$ and $\beta_1 = 2$ and added some Gaussian noise with unit variance. So then when I fit the data the lm function and used the summary function to examine the model I have the following output: \begin{align} \hat \beta_0 & = 1.21054 \quad \text{with Std. Error} = 0.11508, \\ \hat \beta_1 & = 1.87723 \quad \text{with Std. Error} = 0.09844. \end{align}

So how do I interpret the standard error values? For instance, take $\hat \beta_0$, precisely what is $0.11508$ telling me?

Obviously if I ran the simulation a second time, this time adding Gaussian noise with a higher amount of variance, the standard error would increase as the extra variance in the noise shows up as an increase in the standard error of the coefficients. But, if we consider the first simulation in isolation, then what does this value of $0.11508$ mean?

Answer 1

The standard error is the square root of an estimate of the sampling variability of $\hat\beta_j$ as an estimator of $\beta_j$, or $\sqrt{\widehat{Var}(\hat\beta_j)}$.

As this is many things in one sentence, step-by-step:

1. "Square-root": should be self-explanatory, to turn a variance into a standard deviation (that turns out to be what we need in, for example, t-statistics and confidence intervals).
2. "$\hat\beta_j$ as an estimator of $\beta_j$": we use the LS estimator to estimate the unknown parameter $\beta_j$.
3. To do so, we make use of a sample from the underlying population. Had we drawn another sample (or were to draw a fresh one tomorrow, etc.) we would get another estimate $\hat\beta_j$. This is the source of sampling variability. We may summarize that variability through the variance, $Var(\hat\beta_j)$. An expression for this variance may be found, e.g., here.
4. "An estimate of the sampling variability": $Var(\hat\beta_j)$ depends on unknown quantities (like the variance of the Gaussian noise that you generated), which must therefore be estimated, as captured by the formula $\widehat{Var}(\hat\beta_j)$. A formula for this estimator is, for example, given here, or, more introductory, here.

Answer 2

It’s the usual definition of the standard error: the (estimated) standard deviation of the sampling distribution of $\hat{\beta}_0$.

If you were to replicate the work many times with new observations, you would get a distribution of values. Sometimes it would be higher than you observed this time, sometimes lower.

We use the standard error in parameter inference. Being loose, if the p-value on the parameter is less than $0.05$, corresponding to a point estimate about $2$ standard errors above or below $0$, then we might say that the population parameter is not zero, so that variable has a measurable impact on the outcome.

(There are all sorts of caveats about p-values, and discussing them really warrants a separate question (or a master’s degree in statistics).)

Answer 3

If

\begin{align} \hat \beta_0 & = 1.21054 \quad \text{with Std. Error} = 0.11508, \\ \hat \beta_1 & = 1.87723 \quad \text{with Std. Error} = 0.09844. \end{align}

it means that the range of values for the coefficient estimates are

$$\hat \beta_0 = 1.21054 \pm 0.11508$$

and

$$\hat \beta_1 = 1.87723 \pm 0.09844$$

In other words, you can be confident that $\beta_0$ can take on values between $1.09546$ and $1.32562$.

As for your repeated question of

But for a single sample in isolation: what does it tell me when the coefficient is 1.21054 with a standard error of 0.11508

it is not relevant because the $\beta$s (and therefore the $\sigma(\beta)$s) are computed based on the entire sample set, not from one specific observation. A $\beta$ relates the entire $y$ sample set with the entire sample set being input for (one of) the corresponding $x$ vector.

I think what you mean to ask is, what does $\hat \beta_0 = 1.21054 \pm 0.11508$ mean for the output $\hat{y}$ of my fitted model if a new sample $x_i=0.2$ is observed. Well, since $y = \beta_0 x + \beta_0 x + \epsilon$, then the predicted output given that new input is

$$\hat{y}_i = (1.21054 \pm 0.11508) \times 0.2 + (\beta_1 \pm \sigma(\beta_1))\times 0.2 + \epsilon$$

Answer 4

What you created here is a model, that tries to reflect reality. But of course, unless we are exceptionally lucky, the model will never reflect reality perfectly. And the standard deviations reflect how confident the model is about itself.

In your question, you stated you generated data with $\beta_0 = 1$ and $\beta_1=2$. Those numbers are the reality your model tries to reflect. Now suppose you didn't tell us those values, just your model. What can we say about your input?

The model tells us the most likely values are $\beta_0 = 1.21042$ and $\beta_1=1.87223$. But could it be that you that the actual values you put in (the reality) were $1.2$ and $1.9$? Well, therefore we have to look at the standard deviation.

With the given standard deviations, the model tells you it's $68 \%$ sure the true value of $\beta_0$ is in the range $1.09546 - 1.32562$ (minus 1 sd and plus 1 sd). And it's $95 \%$ sure the true value is in the range $0.98038 - 1.4407$ (2 sd away). For $\beta_1$, we can do a similar calculation. That means the numbers $1.2$ and $1.9$ are very reasonable guesses, but that $1$ and $2$ are also not too outlandish.

Now in reality, we often don't have access to the true values of $\beta_0$ and $\beta_1$. We can just take measurements, and make the best model we have. Or sometimes, theorists will come up with a theory that has to be tested on reality, to check whether the model is right or wrong.

As an experimental physicist, you'll run some experiments and maybe get the same values you got. You'll make a model, and can publish this to show that a theory that predicts $\beta_0 = 0$ and $\beta_1=5$ is most definitely wrong (if you can prove your experimental setup is correct). The values you got of $1.21$ and $1.87$ are basically your best guesses as to what the true values could be. But a theory that predicts $\beta_0=1$ and $\beta_1=2$ may well be correct.

Until you come up with an experiment that's more sensitive. Suppose you do the same, and get a model that shows: \begin{align} \hat \beta_0 & = 1.19554 \quad \text{with Std. Error} = 0.01279, \\ \hat \beta_1 & = 1.88341 \quad \text{with Std. Error} = 0.02369. \end{align}

These values allign quite well with your earlier result (showing there was likely no systemic error in your first experiment). But they have much narrower standard deviations, and now also show the theory with $\beta_0=1$ and $\beta_1=2$ is also wrong. But the guesses of $1.2$ and $1.9$ are still holding.