11
$\begingroup$

For a simple linear regression, you get the estimates for the coefficients; however, what exactly is the standard error of the coefficient estimate?

The thing is, if you annotate "standard error" to an entity, that entity has to have many observations ( std error, then is simply the standard deviation). But coefficient estimate for linear regression is calculated by the least squares method, and that will result only in one value of the coefficient. Then what is the standard error of that one single value?

$\endgroup$
1

2 Answers 2

14
$\begingroup$

Let a simple linear regression model

$$ y_i = \beta_1 + \beta_2x_i + \epsilon_i $$ from $n$ observations, where $\epsilon_i$ are iid and of same variance $\sigma^2$.

OLS estimators of $\beta_1$ and $\beta_2$ are given by $$ \hat{\beta}_2 = \frac{\sum(x_i-\bar{x})y_i}{\sum(x_i - \bar{x})^2} $$ and $$ \hat{\beta}_1 = \bar{y} - \hat{\beta}_2 \bar{x} $$ where $\bar{x}$ denotes sample mean. From each parameter we only have one value (since we have one sample). We do not need to estimate $\sigma^2$ to compute both $\hat{\beta_1}$ and $\hat{\beta}_2$.
However, it can be estimated with \begin{align*} \hat{\sigma}^2 &= \frac{1}{n-2} \sum( y_i - \hat{y}_i )^2 \\ &= \frac{1}{n-2} \sum( y_i - \hat{\beta}_1 - \hat{\beta}_2 x_i )^2 \end{align*}

But even if we only have one value for each estimator, we have the following results : for OLS estimates $\hat{\beta}_1$ and $\hat{\beta}_2$.

$$ \text{Var}(\hat{\beta}_1) =\frac{1}{n} \frac{\sigma^2 \sum x_i^2}{\sum(x_i - \bar{x})^2} $$ and $$ \text{Var}(\hat{\beta}_2) = \frac{\sigma^2}{\sum(x_i - \bar{x})^2} $$

Moreover,

$$ \text{Cov}(\hat{\beta}_1,\hat{\beta}_2) = - \frac{\sigma^2 \bar{x}}{\sum(x_i - \bar{x})^2} $$ Proof of these results can be found in any textbook on linear regression.

Since we can estimate $\sigma^2$ from the original sample, we can estimate variances of estimators, even if we only have one value of them.

These variances estimates can also be computed from bootstrapping the sample, i.e by taking samples of the original sample and by computing estimators for each sub sample. For example for $\beta_1$, if you take $K$ sub sample then you get a sample $\hat{\beta}_1^{(1)},\dots,\hat{\beta}_1^{(K)}$ from which you can empirically estimate the variance of $\hat{\beta}_1$.

Here is a simple code (in R) showing how the two methods can be used

data <- data.frame(do.call(rbind, lapply(1:500, function(i){
    id=i
    x <- rexp(1,1)
    y <- 1 + 3*x + rnorm(1,0,1)
    return(c(id,y,x))
})))
names(data) <- c("id", "y","x")

summary(lm(y ~ x, data)) ## OLS std estimates from full sample

## by bootstrapping
Nboot <- 500
list <- lapply(1:Nboot, function(i){
    Ids <- sort(sample(data$id, replace=TRUE))
data.s = data[Ids,]
mod.s <- lm(y ~ x, data.s)
return(mod.s$coefficients)
})
## Bootstrap variances estimates
var(sapply(list,function(x) x[[1]]))**0.5  # beta1
var(sapply(list,function(x) x[[2]]))**0.5  # beta2

in the command

summary(lm(y ~ x, data)) ## OLS std estimates

the values in the "Std. Error" column should be close to the values computed in the last two lines

$\endgroup$
1
  • 2
    $\begingroup$ This is a great answer, and really helped me. However, I think it would benefit from elaborating the passage from $\hat{\sigma}^2$ to $\text{Var}(\hat{\beta}_1)$ and $\text{Var}(\hat{\beta}_2)$. I cannot see how that passage might work. $\endgroup$
    – alelom
    Commented May 8, 2020 at 9:47
12
$\begingroup$

I think OP may be more concerned with an intuitive understanding of a standard error than of its calculation.

Consider a population. In the population, there is a population slope $\beta_1$ and a population intercept $\beta_0$ that govern the relationship between $X$ and $Y$. If you draw a sample from that population, using the formulas described in @winperikle's answer, you can compute the sample slope and sample intercept, $\hat{\beta}_1$ and $\hat{\beta}_0$. They won't be exactly equal to $\beta_1$ and $\beta_0$ due to sampling error (i.e., random chance), but they will probably be close (especially with a large sample).

Imagine we could do the previous activity again: draw a sample from the population and compute the sample slope and sample intercept, $\hat{\beta}_1$ and $\hat{\beta}_0$ (but we'll just focus on the sample slope for now). Imagine repeating this activity an infinite amount of times. You now have a collection of $\hat{\beta}_1$s, one from each sample. These will not all be equal to each other across samples because each sample is different. We can say that the many values of $\hat{\beta}_1$ have a distribution which has some variability and a center.

The variability of the collection of $\hat{\beta}_1$s can be quantified as the standard error, which is simply the standard deviation of the $\hat{\beta}_1$s. It is a fixed value that depends only on the qualities of the population and the size of each of the samples. If we knew this value, we could easily compute how far a single sample slope $\hat{\beta}_1$ is from the value of $\beta_1$ under the null hypothesis, which would allow us to determine whether we want to reject or fail to reject the null hypothesis.

In practice, we don't have access to the true standard error. We get to draw one sample and try to extract as much information as possible from it. The formula for the standard error of $\hat{\beta}_1$ in @winperikle's answer is a good estimate of the true standard error, even though we're only calculating it from one sample. So, from our one sample, we can compute an estimate of $\beta_1$ and an estimate of the standard error. This is where the standard error in regression output comes from.

$\endgroup$
3
  • 1
    $\begingroup$ great explanation ! But , I have a question . If we use bootstrapping to calculate the standard error of the estimators , how is the variance of the prediction error , (sigma)^2 found ? Is the prediction error found for each bootstrap sample and then the overall variance across the samples is used in the formula of standard error of the estimator ? $\endgroup$
    – Bharathi
    Commented Jul 8, 2020 at 18:09
  • 1
    $\begingroup$ I'm not sure about calculating specific values. My answer was primarily about the interpretation of standard errors, but I can't say much about how specific quantities are computed. Just to be clear, my answer was not about bootstrapping; it was about the theoretical construction of a sampling distribution. $\endgroup$
    – Noah
    Commented Jul 9, 2020 at 1:35
  • $\begingroup$ When using bootstrap to compute a sampling distribution you 1) draw resamples 2) on each resample compute your estimated $\beta_1$, $\beta_2$ (i.e. you have this e.g. 1000 times) and finally you 3) use the estimator of the variance over this set of beta_1,2 to estimate the variance of beta_1,2. Like this you do not even have to know the formulas in above post. You can bootstrap your way out without any distribution assumption. Great!! $\endgroup$
    – Ggjj11
    Commented Oct 2 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.