I am reading a book in which it gives a linear regression example with multiple features, and it talks about standard error of the estimate (SE) for the weights of each feature, as shown below First column is each feature (some of the one-hot-encoded)

but from what I understand, SE has to be a single number and has nothing to do with features but rather has to do with the predicted values and the actual values (ground truths). thanks in advance and hope my description is clear

  • 2
    $\begingroup$ The standard error is a single number. These standard errors refer to the sampling distribution of the coefficients in the linear model. A standard error of a statistic is usually used to describe the variance of that statistic's sampling distribution, so it doesn't always have to be about predictions. $\endgroup$ – Demetri Pananos Oct 30 '19 at 20:49
  • $\begingroup$ Thanks. After searching for "sampling distribution of the regression coefficients" I now understand how the values are calculated but still I am not sure I understand what these distributions signify. is it just like, for instance, the (normal) distribution of the height of 100 people chosen from a population, but for the possible weights/coefficients of the linear regression? $\endgroup$ – Ash Oct 31 '19 at 20:20
  • $\begingroup$ Imagine drawing 100 different datasets from the same population and fitting 100 models. The sampling distribution is the distribution of those 100 coefficients. $\endgroup$ – Demetri Pananos Oct 31 '19 at 22:49

Here is a little function which will essentially run a regression and return a coefficient. The data on which we will perform the regression are always generated from the same data generating process (same coefficients, every time).'

  N = 50
  x = rnorm(N)
  y = 2*x+1+rnorm(N)

  model = lm(y~x)
  b = coef(model)[2]

To see what the standard errors for the coefficients really mean, let's run this function 10,000 times and plot a histogram.

rerun(10000, sim()) %>% 
  unlist() %>% 
  tibble(b= .) %>% 
  geom_histogram(color = 'white')

Our result looks like

enter image description here

This histogram is made from collecting coefficients from our modelling process 10,000 times. This distribution has some variance. It's about 0.144 depending on the random seed you use.

Now let's take a look at the standard errors for our model coefficient.

lm(formula = y ~ x)

     Min       1Q   Median       3Q      Max 
-1.92989 -0.64283 -0.05236  0.61998  1.99187 

            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   1.1151     0.1326    8.41 5.33e-11 ***
x             2.1476     0.1443   14.88  < 2e-16 ***
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.9372 on 48 degrees of freedom
Multiple R-squared:  0.8219,    Adjusted R-squared:  0.8182 
F-statistic: 221.5 on 1 and 48 DF,  p-value: < 2.2e-16

Note that the standard error for the coefficient of x is 0.144.

That is what the standard error refers to in regression. It is the variance of the distribution of coefficients were I to collect data and run a model over and over and over.

  • $\begingroup$ this is great. thanks $\endgroup$ – Ash Oct 31 '19 at 23:36
  • $\begingroup$ @Demetri Pananos, thank you for an intelligible description! $\endgroup$ – Heather Kaye Nov 1 '19 at 8:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.