I'm beginning to look at tables more and more in my studies, but I don't understand the significance of the standard errors below the coefficient estimates. I know if you divide the estimate by the s.e. you get a tstat which provides a test for significance, but it seems like my professor can just look at it and determine at what level it is significant.

Can someone provide a simple way to interpret the s.e. on a regression table? Am I missing something?

Edit : This has been a great discussion and I'm going to digest some of the information before commenting further and deciding on an answer. Thank you for all your responses.

  • Doesn't the thread at stats.stackexchange.com/questions/5135/… address this question? If you are concerned with understanding standard errors better, then looking at some of the top hits in a site search may be helpful. – whuber Dec 3 '14 at 20:53
  • 2
    If your n's are large, all your professor is likely doing is comparing the ratio Est/se to a handful of z-values that he or she has memorized e.g. (1.645, 1.96, 2.58, 3.29). That's nothing amazing - after doing a few dozen such tests, that stuff should be straightforward. – Glen_b Dec 3 '14 at 22:47
  • The value of the coefficient has to be double of standard error (s.e)2 for 5% significate level. – user218026 yesterday
up vote 3 down vote accepted

The standard error determines how much variability "surrounds" a coefficient estimate. A coefficient is significant if it is non-zero. The typical rule of thumb, is that you go about two standard deviations above and below the estimate to get a 95% confidence interval for a coefficient estimate.

So most likely what your professor is doing, is looking to see if the coefficient estimate is at least two standard errors away from 0 (or in other words looking to see if the standard error is small relative to the coefficient value). This is how you can eyeball significance without a p-value.

  • 2
    "A coefficient is significant" if what is nonzero? The standard error? The variability? The coefficient? (Since none of those are true, it seems something is wrong with your assertion. I don't question your knowledge, but it seems there is a serious lack of clarity in your exposition at this point.) – whuber Dec 3 '14 at 20:54
  • 1
    @whuber For clarity, perhaps it should be restated as: "A variable is significant if it's corresponding coefficient estimate is significantly different from zero." – Underminer Dec 3 '14 at 21:52
  • That sounds rather circular to me--you are trying to describe or explain "significant" in terms of "significantly different." – whuber Dec 3 '14 at 21:57
  • @whuber Can you explain why that sounds circular? If a variable's coefficient estimate is significantly different from zero (or some other null hypothesis value), then the corresponding variable is said to be significant. It seems like simple if-then logic to me. – Underminer Dec 3 '14 at 22:16
  • 1
    @Underminer thanks for this clarification. I went back and looked at some of my tables and can see what you are talking about now. Two S.D. for 95% confidence, and one S.D. for 90%? – Amstell Dec 3 '14 at 23:01

I will stick to the case of a simple linear regression. Generalisation to multiple regression is straightforward in the principles albeit ugly in the algebra. Imagine we have some values of a predictor or explanatory variable, $x_i$, and we observe the values of the response variable at those points, $y_i$. If the true relationship is linear, and my model is correctly specified (for instance no omitted-variable bias from other predictors I have forgotten to include), then those $y_i$ were generated from:

$$y_i = \beta_0 + \beta_1 x_i + \epsilon_i$$

Now $\epsilon_i$ is random error or disturbance term, which has, let's say, the $\mathcal{N}(0,\sigma^2)$ distribution. That assumption of normality, with the same variance (homoscedasticity) for each $\epsilon_i$, is important for all those lovely confidence intervals and significance tests to work. For the same reason I shall assume that $\epsilon_i$ and $\epsilon_j$ are not correlated so long as $i \neq j$ (we must permit, of course, the inevitable and harmless fact that $\epsilon_i$ is perfectly correlated with itself) - this is the assumption that disturbances are not autocorrelated.

Note that all we get to observe are the $x_i$ and $y_i$, but that we can't directly see the $\epsilon_i$ and their $\sigma^2$ or (more interesting to us) the $\beta_0$ and $\beta_1$. We obtain (OLS or "least squares") estimates of those regression parameters, $\hat{\beta_0}$ and $\hat{\beta_1}$, but we wouldn't expect them to match $\beta_0$ and $\beta_1$ exactly. Moreover, if I were to go away and repeat my sampling process, then even if I use the same $x_i$'s as the first sample, I won't obtain the same $y_i$'s - and therefore my estimates $\hat{\beta_0}$ and $\hat{\beta_1}$ will be different to before. This is because in each new realisation, I get different values of the error $\epsilon_i$ contributing towards my $y_i$ values.

The fact that my regression estimators come out differently each time I resample, tells me that they follow a sampling distribution. If you know a little statistical theory, then that may not come as a surprise to you - even outside the context of regression, estimators have probability distributions because they are random variables, which is in turn because they are functions of sample data that is itself random. With the assumptions listed above, it turns out that:

$$\hat{\beta_0} \sim \mathcal{N}\left(\beta_0,\, \sigma^2 \left( \frac{1}{n} + \frac{\bar{x}^2}{\sum(X_i - \bar{X})^2} \right) \right) $$

$$\hat{\beta_1} \sim \mathcal{N}\left(\beta_1, \, \frac{\sigma^2}{\sum(X_i - \bar{X})^2} \right) $$

It's nice to know that $\mathbb{E}(\hat{\beta_i}) = \beta_i$, so that "on average" my estimates will match the true regression coefficients (actually this fact doesn't need all the assumptions I made before - for instance it doesn't matter if the error term is not normally distributed, or if they're heteroscedastic, but correct model specification with no autocorrelation of errors is important). If I were to take many samples, the average of the estimates I obtain would converge towards the true parameters. You may find this less reassuring once you remember that we only get to see one sample! But the unbiasedness of our estimators is a good thing.

Also interesting is the variance. In essence this is a measure of how badly wrong our estimators are likely to be. For example, it'd be very helpful if we could construct a $z$ interval that lets us say that the estimate for the slope parameter, $\hat{\beta_1}$, we would obtain from a sample is 95% likely to lie within approximately $\pm 1.96 \sqrt{\frac{\sigma^2}{\sum(X_i - \bar{X})^2}}$ of the true (but unknown) value of the slope, $\beta_1$. Sadly this is not as useful as we would like because, crucially, we do not know $\sigma^2$. It's a parameter for the variance of the whole population of random errors, and we only observed a finite sample.

If instead of $\sigma$ we use the estimate $s$ we calculated from our sample (confusingly, this is often known as the "standard error of the regression" or "residual standard error") we can find the standard error for our estimates of the regression coefficients. For $\hat{\beta_1}$ this would be $\sqrt{\frac{s^2}{\sum(X_i - \bar{X})^2}}$. Now, because we have had to estimate the variance of a normally distributed variable, we will have to use Student's $t$ rather than $z$ to form confidence intervals - we use the residual degrees of freedom from the regression, which in simple linear regression is $n-2$ and for multiple regression we subtract one more degree of freedom for each additional slope estimated. But for reasonably large $n$, and hence larger degrees of freedom, there isn't much difference between $t$ and $z$. Rules of thumb like "there's a 95% chance that the observed value will lie within two standard errors of the correct value" or "an observed slope estimate that is four standard errors away from zero will clearly be highly statistically significant" will work just fine.

I find a good way of understanding error is to think about the circumstances in which I'd expect my regression estimates to be more (good!) or less (bad!) likely to lie close to the true values. Suppose that my data were "noisier", which happens if the variance of the error terms, $\sigma^2$, were high. (I can't see that directly, but in my regression output I'd likely notice that the standard error of the regression was high.) Then most of the variation I can in $y$ see will be due to the random error. This will mask the "signal" of the relationship between $y$ and $x$, which will now explain a relatively small fraction of variation, and makes the shape of that relationship harder to ascertain. Note that this does not mean I will underestimate the slope - as I said before, the slope estimator will be unbiased, and since it is normally distributed, I'm just as likely to underestimate as I am to overestimate. But since it is harder to pick the relationship out from the background noise, I am more likely than before to make big underestimates or big overestimates. My standard error has increased, and my estimated regression coefficients are less reliable.

Intuition matches algebra - note how $s^2$ appears in the numerator of my standard error for $\hat{\beta_1}$, so if it's higher, the distribution of $\hat{\beta_1}$ is more spread out. This means more probability in the tails (just where I don't want it - this corresponds to estimates far from the true value) and less probability around the peak (so less chance of the slope estimate being near the true slope). Here is are the probability density curves of $\hat{\beta_1}$ with high and low standard error:

Probably density of regression slope estimator with high and low standard error

It's instructive to rewrite the standard error of $\hat{\beta_1}$ using the mean square deviation, $$\text{MSD}(x) = \frac{1}{n} \sum(x_i - \bar{x})^2$$ This is a measure of how spread out the range of observed $x$ values was. With this in mind, the standard error of $\hat{\beta_1}$ becomes:

$$\text{se}(\hat{\beta_1}) = \sqrt{\frac{s^2}{n \text{MSD}(x)}}$$

The fact that $n$ and $\text{MSD}(x)$ are in the denominator reaffirms two other intuitive facts about our uncertainty. We can reduce uncertainty by increasing sample size, while keeping constant the range of $x$ values we sample over. As ever, this comes at a cost - that square root means that to halve our uncertainty, we would have to quadruple our sample size (a situation familiar from many applications outside regression, such as picking a sample size for political polls). But it's also easier to pick out the trend of $y$ against $x$, if we spread our observations out across a wider range of $x$ values and hence increase the MSD. Again, by quadrupling the spread of $x$ values, we can halve our uncertainty in the slope parameters.

When you chose your sample size, took steps to reduce random error (e.g. from measurement error) and perhaps decided on the range of predictor values you would sample across, you were hoping to reduce the uncertainty in your regression estimates. In that respect, the standard errors tell you just how successful you have been.

I append code for the plot:

x <- seq(-5, 5, length=200)
y <- dnorm(x, mean=0, sd=1)
y2 <- dnorm(x, mean=0, sd=2)
plot(x, y, type = "l", lwd = 2, axes = FALSE, xlab = "estimated coefficient", ylab="")
lines(x, y2, lwd = 2, col = "blue")
axis(1, at = c(-5, -2.5, 0, 2.5, 5), labels = c("", "large underestimate", "true β", "large overestimate", ""))
abline(v=0, lty = "dotted")
legend("topright", title="Standard error of estimator",
   c("Low","High"), fill=c("black", "blue"), horiz=TRUE)

The SE is a measure of precision of the estimate. It also can indicate model fit problems. For example, if it is abnormally large relative to the coefficient then that is a red flag for (multi)collinearity. The model is essentially unable to precisely estimate the parameter because of collinearity with one or more of the other predictors.

The SE is essentially the standard deviation of the sampling distribution for that particular statistic. This is why a coefficient that is more than about twice as large as the SE will be statistically significant at p=<.05. You might go back and look at the standard deviation table for the standard normal distribution (Wikipedia has a nice visual of the distribution).

Think of it this way, if you assume that the null hypothesis is true - that is, assume that the actual coefficient in the population is zero, how unlikely would your sample have to be in order to get the coefficient you got? If your sample statistic (the coefficient) is 2 standard errors (again, think "standard deviations") away from zero then it is one of only 5% (i.e. p=.05) of samples that are possible assuming that the true value (the population parameter) is zero. That's is a rather improbable sample, right? So we conclude instead that our sample isn't that improbable, it must be that the null hypothesis is false and the population parameter is some non zero value. We "reject the null hypothesis." Hence, the statistic is "significant" when it is 2 or more standard deviations away from zero which basically means that the null hypothesis is probably false because that would entail us randomly picking a rather unrepresentative and unlikely sample.

I am playing a little fast and lose with the numbers. There is, of course, a correction for the degrees freedom and a distinction between 1 or 2 tailed tests of significance. With a good number of degrees freedom (around 70 if I recall) the coefficient will be significant on a two tailed test if it is (at least) twice as large as the standard error. With a 1 tailed test where all 5% of the sampling distribution is lumped in that one tail, those same 70 degrees freedom will require that the coefficient be only (at least) ~1.7 times larger than the standard error.

So twice as large as the coefficient is a good rule of thumb assuming you have decent degrees freedom and a two tailed test of significance. Less than 2 might be statistically significant if you're using a 1 tailed test. More than 2 might be required if you have few degrees freedom and are using a 2 tailed test.

edited to add: Something else to think about: if the confidence interval includes zero then the effect will not be statistically significant. The confidence interval (at the 95% level) is approximately 2 standard errors. Confidence intervals and significance testing rely on essentially the same logic and it all comes back to standard deviations.

If you can divide the coefficient by its standard error in your head, you can use these rough rules of thumb assuming the sample size is "large" and you don't have "too many" regressors. When this is not the case, you should really be using the $t$ distribution, but most people don't have it readily available in their brain.

These rules are derived from the standard normal approximation for a two-sided test ($H_0: \beta=0$ vs. $H_a: \beta\ne0$)):

  • 1.28 will give you SS at $20\%$.

  • 1.64 will give you SS at $10\%$

  • 1.96 will give you SS at $5\%$

  • 2.56 will give you SS at $1\%$

SS is shorthand for "statistically significant from zero in a two-sided test".

Often, you will see the 1.96 rounded up to 2.

  • These rules appear to be rather fussy--and potentially misleading--given that in most circumstances one would want to refer to a Student t distribution rather than a Normal distribution. It should suffice to remember the rough value pairs $(5/100, 2)$ and $(2/1000, 3)$ and to know that the second value needs to be substantially adjusted upwards for small sample sizes (less than $20$ or so). – whuber Dec 4 '14 at 0:12
  • @whuber I have never seen a regression with a sample size that small in a class, but I will add that caveat. – Dimitriy V. Masterov Dec 4 '14 at 0:21

Picking up on Underminer, regression coefficients are estimates of a population parameter. Due to sampling error (and other things if you have accounted for them), the SE shows you how much uncertainty there is around your estimate. If you calculate a 95% confidence interval using the standard error, that will give you the confidence that 95 out of 100 similar estimates will capture the true population parameter in their intervals. Just another way of saying the p value is the probability that the coefficient is do to random error.

Also, SEs are useful for doing other hypothesis tests - not just testing that a coefficient is 0, but for comparing coefficients across variables or sub-populations.

  • 2
    You were doing great until the last line of the first paragraph. Are you really claiming that a large p-value would imply the coefficient is likely to be "due to random error"? Indeed, given that the p-value is the probability for an event conditional on assuming the null hypothesis, if you don't know for sure whether the null is true, then why would it make any sense to interpret the p-value as an actual (rather than purely hypothetical) probability? – whuber Dec 3 '14 at 20:51
  • Excellent point - one many statisticians (myself included) I work with ignore. – robin.datadrivers Dec 3 '14 at 21:15

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.