0
$\begingroup$

dear members, I have been troubling myself with this question for the past few days but have not found any answers on the Internet:

For a simple linear regression, you get the estimates for the coefficients; however, what exactly is the standard error of the coefficient estimate?

The thing is, if you annotate "standard error" to an entity, that entity has to have many observations ( std error, then is simply the standard deviation). But coefficient estimate for linear regression is calculated by the least squares method, and that will result ONLY IN ONE VALUE OF THE COEFFICIENT. Then what is the standard error of that ONE SINGLE VALUE???!!!!!!!!

$\endgroup$
2
$\begingroup$

Let a simple linear regression model

$$ y_i = \beta_1 + \beta_2x_i + \epsilon_i $$ from $n$ observations, where $\epsilon_i$ are iid and of same variance $\sigma^2$.

OLS estimators of $\beta_1$ and $\beta_2$ are given by $$ \hat{\beta}_2 = \frac{\sum(x_i-\bar{x})y_i}{\sum(x_i - \bar{x}^2} $$ and $$ \hat{\beta}_1 = \bar{y} - \hat{\beta}_2 \bar{x} $$ where $\bar{x}$ denotes sample mean. From each parameter we only have one value (since we have one sample). We do not need to estimate $\sigma^2$ to compute both $\hat{\beta_1}$ and $\hat{\beta}_2$.
However, it can be estimated with \begin{align*} \hat{\sigma}^2 &= \frac{1}{n-2} \sum( y_i - \hat{y}_i )^2 \\ &= \frac{1}{n-2} \sum( y_i - \hat{\beta}_1 - \hat{\beta}_2 x_i )^2 \end{align*}

But even if we only have one value for each estimator, we have the following results : for OLS estimates $\hat{\beta}_1$ and $\hat{\beta}_2$.

$$ \text{Var}(\hat{\beta}_1) =\frac{1}{n} \frac{\sigma^2 \sum x_i^2}{\sum(x_i - \bar{x})^2} $$ and $$ \text{Var}(\hat{\beta}_2) = \frac{\sigma^2}{\sum(x_i - \bar{x})^2} $$

Moreover,

$$ \text{Cov}(\hat{\beta}_1,\hat{\beta}_2) = - \frac{\sigma^2 \bar{x}}{\sum(x_i - \bar{x})^2} $$ Proof of these results can be found in any textbook on linear regression.

Since we can estimate $\sigma^2$ from the original sample, we can estimate variances of estimators, even if we only have one value of them.

These variances estimates can also be computed from bootstrapping the sample, i.e by taking samples of the original sample and by computing estimators for each sub sample. For example for $\beta_1$, if you take $K$ sub sample then you get a sample $\hat{\beta}_1^{(1)},\dots,\hat{\beta}_1^{(K)}$ from which you can empirically estimate the variance of $\hat{\beta}_1$.

Here is a simple code (in R) showing how the two methods can be used

data<-data.frame(do.call(rbind,lapply(1:500,function(i){
id=i
x<-rexp(1,1)
y<- 1 + 3*x + rnorm(1,0,1)
return(c(id,y,x))
})))
names(data)<-c("id", "y","x")



summary(lm(y~x,data)) ## OLS std estimates from full sample


## by bootstrapping
Nboot<-500
list<-lapply(1:Nboot,function(i){
Ids <- sort(sample(data$id,replace=TRUE))
data.s = data[Ids,]
mod.s<-lm(y~x,data.s)
return(mod.s$coefficients)
})
## Bootstrap variances estimates
var(sapply(list,function(x) x[[1]]))**0.5  # beta1
var(sapply(list,function(x) x[[2]]))**0.5  # beta2

in the command

summary(lm(y~x,data)) ## OLS std estimates

the values in the "Std. Error" column should be close to the values computed in the last two lines

$\endgroup$
0
$\begingroup$

I think OP may be more concerned with an intuitive understanding of a standard error than of its calculation.

Consider a population. In the population, there is a population slope $\beta_1$ and a population intercept $\beta_0$ that govern the relationship between $X$ and $Y$. If you draw a sample from that population, using the formulas described in @winperikle's answer, you can compute the sample slope and sample intercept, $\hat{\beta}_1$ and $\hat{\beta}_0$. They won't be exactly equal to $\beta_1$ and $\beta_0$ due to sampling error (i.e., random chance), but they will probably be close (especially with a large sample).

Imagine we could do the previous activity again: draw a sample from the population and compute the sample slope and sample intercept, $\hat{\beta}_1$ and $\hat{\beta}_0$ (but we'll just focus on the sample slope for now). Imagine repeating this activity an infinite amount of times. You now have a collection of $\hat{\beta}_1$s, one from each sample. These will not all be equal to each other across samples because each sample is different. We can say that the many values of $\hat{\beta}_1$ have a distribution which has some variability and a center.

The variability of the collection of $\hat{\beta}_1$s can be quantified as the standard error, which is simply the standard deviation of the $\hat{\beta}_1$s. It is a fixed value that depends only on the qualities of the population and the size of each of the samples. If we knew this value, we could easily compute how far a single sample slope $\hat{\beta}_1$ is from the value of $\beta_1$ under the null hypothesis, which would allow us to determine whether we want to reject or fail to reject the null hypothesis.

In practice, we don't have access to the true standard error. We get to draw one sample and try to extract as much information as possible from it. The formula for the standard error of $\hat{\beta}_1$ in @winperikle's answer is a good estimate of the true standard error, even though we're only calculating it from one sample. So, from our one sample, we can compute an estimate of $\beta_1$ and an estimate of the standard error. This is where the standard error in regression output comes from.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.