3
$\begingroup$

I am quite confused in these terminologies (especially but not limited to regression)
I do understand what Variance and Standard Deviation means, they measure the dispersion / variability of the data.


However, according to my understanding, Standard Error $= \frac{s}{\sqrt{n}}$ where s is the sample standard deviation.
But in regression (for simplicity, here refer to Simple Linear Regression but MLR shall be of the same fashion) $y = \beta_0 + \beta_{1}x + e$.
Variance of $\hat\beta_1$ = $\frac{\sigma^2}{SXX}$
And while we are doing confidence interval for $\hat\beta_1$, the SE we use is simply the square root of Var($\hat\beta_1$) = $\frac{\sigma}{\sqrt{SXX}}$ without needing to divide by $\sqrt{n}$

My questions:
1) Is there a difference between normal Standard Error (of mean) that we talk about (i.e. $\frac{s}{\sqrt{n}}$) and the SE we talk in regression.
2) I suppose, $\hat\beta_1$ is not a mean but purely an estimator of the parameter $\beta_1$, so why do we use SE when we are constructing confidence interval of $\hat\beta_1$?
3) What about confidence interval for predicted $y$ value and fitted $y$ value respectively?

$\endgroup$
1
  • $\begingroup$ Whenever you speak of the standard deviation of an estimator that is a standard error. For example $\bar X$ is an estimator of $\mu$ so $SD(\bar X) = SE(\bar X) = \sigma/\sqrt{n}.$ Often, if $\sigma$ is unknown and estimated by $S$ authors say that the 'standard error' of $\bar X$ is $S/\sqrt{n}$ when more fastidious terminology would be 'estimated standard error'. (The word estimated gets dropped, usually with no harm, because one knows $S$ is an estimate.) $\endgroup$ – BruceET Nov 5 '20 at 16:30
2
$\begingroup$

The term "standard error" refers to the standard deviation of a statistic that is calculated. So, you can calculate a standard error for a mean--because the mean is a statistic. You can also calculate a standard error for a parameter estimate like $\hat{\beta}$.

We say standard error instead of standard deviation to distinguish between a value that's calculated from repeated observations and an estimate that's based on a theory about the distribution.

We only have one observation for $\hat{\beta}$, and we have mathematical theory to derive its sampling error--so we call that the standard error.

We have more than one observation of a variable X, and we calculate the sampling error based on that observed data--so we call that statistic the standard deviation.

$\endgroup$
3
  • $\begingroup$ so the formula $\frac{s}{\sqrt{n}}$ is just the formula for standard error of the mean ($\bar{x}$) $\endgroup$ – dust Nov 5 '20 at 16:41
  • $\begingroup$ and the difference between standard deviation and standard error is the number of observations. If the data is a statistic, which has only one observation, summarizing a bunch of data point (like the mean summarize the whole set of data) then it will be called standard error. While if it is a lot of observations (like the set of data itself), then it will be the standard deviation? $\endgroup$ – dust Nov 5 '20 at 16:44
  • $\begingroup$ Yes--That's the gist of the reasoning behind having two different terms for what looks like the same thing. $\endgroup$ – Melinda Thielbar PhD Nov 18 '20 at 18:36
2
$\begingroup$

The terminology is the same everywhere in statistics I think:

  • Variance $\sigma^2$ is the second moment of a known probability distribution
  • Standard Deviation $\sigma$ is the square root of variance
  • Variance of the mean $\sigma^2_{\mu} = \frac{\sigma^2}{N}$ is the variance of the mean of $N$ i.i.d random variables
  • Standard Deviation of the Mean $\sigma_{\mu}$ is the square root of the variance of the mean

The 4 above metrics apply analytically to probability distributions. One can estimate any one of them, typically denoted by letter $s$ and prefix 'sample', such as 'sample error of the mean' $s_{\mu}$. Sample standard deviation and Sample standard deviation of the mean are also known as Standard Error and Standard Error of the mean (SEM) respectively

With respect to your questions:

  • Variance and standard deviation are metrics of the distribution of the random variables in analytic case and a metric of data in the sample case. These terms are not applicable to parameters of your model, such as $\beta$ or $\hat \beta$. These are simply the parameter and its estimate.
  • When you construct a confidence interval for an unknown parameter, you perform a hypothesis test. The confidence interval is likely to be a function of the moments of the distribution, or their sample counterparts, but that depends strongly on the underlying distribution.
  • Confidence intervals only apply to unknown parameters of the model, they do not apply to parts of data such as $y$. The closest entity to a confidence interval when applied to random variable itself is a tolerance interval, namely, the interval where the random variable is likely to fall given the exact model parameters
$\endgroup$
3
  • $\begingroup$ Standard deviation and standard error are not the same thing. $\endgroup$ – Eoin Nov 5 '20 at 16:24
  • $\begingroup$ @Eoin I will correct, no need to rush with minuses :). I am used to people using the termis interchangingly $\endgroup$ – Aleksejs Fomins Nov 5 '20 at 16:26
  • $\begingroup$ @Eoin Is it correct now? $\endgroup$ – Aleksejs Fomins Nov 5 '20 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.