0
$\begingroup$

This is an elementary question, but I'm a bit confused on the concept of standard error.

On some sources, I've seen that standard error is defined as the standard deviation or the square root of the variance.

On other sources, I've seen standard error defined as the standard deviation divided by the square root of the sample size.

When is each one used? From my understanding, the former is used when computing the standard error for a random variable, and the latter is used when computing the standard error of some number of samples. In addition, the former involves probability of each event a random variable can take on whereas the latter does not involve probability. Is this correct?

Improve this question
$\endgroup$
6
  • $\begingroup$ See stats.stackexchange.com/a/18609/919. $\endgroup$
    – whuber ♦
    Commented Apr 1, 2020 at 20:10
  • $\begingroup$ the latter is the standard error of the mean $\endgroup$
    – gunes
    Commented Apr 1, 2020 at 20:10
  • $\begingroup$ @gunes What is the former called? Standard error of a random variable? $\endgroup$
    – roulette01
    Commented Apr 1, 2020 at 20:17
  • 1
    $\begingroup$ I did not define the standard error in terms of a formula: I provided the concept. The formulas all depend on the probability model and the statistic. $\endgroup$
    – whuber ♦
    Commented Apr 1, 2020 at 21:44
  • 1
    $\begingroup$ It is difficult to know how to answer your question with no clues about the level of your course. You give no context in your question. And your profile for the site says nothing about your current interests or level of study. $\endgroup$
    – BruceET
    Commented Apr 1, 2020 at 22:27

1 Answer 1

Reset to default
2
$\begingroup$

Here simple examples, in which I focus on the terminology that seems to be confusing you.

Suppose you have a sample of size 100 from the distribution $\mathsf{Norm}(\mu=200, \sigma=25).$

Then the standard deviation of a single observation is $SD(X_i) = \sigma = 25.$ The standard deviation of $\bar X,$ also called the 'standard error of the (sample) mean', is $SD(\bar X) = \sigma/\sqrt{n} = 25/10 = 2.5.$

The usual estimated standard deviation of single observation $X_i$ from the population is $S = \sqrt{\frac{\sum_{i=1}^n (X_i - \bar X)^2}{n-1}}.$

[Notice that $E(S^2) = \sigma^2,$ but $E(S)$ is not exactly equal to $\sigma.$ It's very close for large $n$ and in practice the small discrepancy is usually ignored. If $n = 5,$ for example, then $E(S) \approx 0.940\sigma.$ But if $n = 50,$ we have $E(S) \approx 0.995\sigma.)$

Accordingly, the usual '(estimated) standard error' of the sample mean is $S/\sqrt{n},$ where the word estimated is often omitted for brevity (because as soon as you realize you're using the estimated SD $S$ you have to know the standard error is also an estimate).

If $\mu$ is unknown and $\sigma$ is known, then a 95% confidence interval for $\mu$ is of the form $$\bar X \pm 1.96\frac{\sigma}{\sqrt{n}},$$ where the last factor $\frac{\sigma}{\sqrt{n}}$ in the margin of error is the (exact) standard error of the sample mean $\bar X.$

[The amount $1.96\frac{\sigma}{\sqrt{n}}$ added to and subtracted from $\bar X$ is the 'margin of error' of the confidence interval. Notice that it becomes smaller as $n$ increases. A CI based on $n = 400$ observations is half as long and a CI based on $n = 100$ observations from the same distribution.]

By contrast, if both $\mu$ and $\sigma$ are unknown, then a 95% CI for $\mu$ is of the form

$$\bar X \pm t^*\frac{S}{\sqrt{n}},$$ where $t^*$ cuts probability 0.025 from the upper tail of Student's t distribution with $n-1$ degrees of freedom and the (estimated) standard error of $\bar X$ is $\frac{S}{\sqrt{n}}.$

[In this case, it is also possible to get a confidence interval for $\sigma$ by using a chi-squared distribution, but this confidence interval is of an entirely different form: It is not symmetrical about $S$ and it does not use a 'standard error' of $S.]$

Improve this answer
$\endgroup$
6
  • $\begingroup$ Is the terminology "standard deviation of a single observation" valid? I'm tempted to say the standard deviation of a single observation as zero (because I'd just consider a single sample). Would it be more correct to say "standard deviation of a random variable?" $\endgroup$
    – roulette01
    Commented Apr 1, 2020 at 21:32
  • $\begingroup$ Also, if $E[s^2] = \sigma^2$, why is $E[s] \neq \sigma$? Shouldn't both be $\neq$? $\endgroup$
    – roulette01
    Commented Apr 1, 2020 at 21:36
  • $\begingroup$ OK then 'standard deviation of a randomly sampled observation'. // $E(aX + b) = aE(X) + b,$ for random variable $X$ and constants $a$ and $b.$ However, expectation is a 'linear operator' because of the way it is defined. So equality does not survive nonlinear operations like taking the square root. $\endgroup$
    – BruceET
    Commented Apr 1, 2020 at 21:43
  • $\begingroup$ I'm confused why $E[S^2] = \sigma^2$ exactly. Could you explain? $\endgroup$
    – roulette01
    Commented Apr 1, 2020 at 21:47
  • $\begingroup$ Maybe you know the general formula $Var(X) = E(X^2) - [E(X)]^2.$ Consider $X \sim \mathsf{Norm}(\mu = 100, \sigma=15).$ Then $E(X^2) \ne [E(X)]^2.$ Instead $E(X^2) = \sigma^2 + \mu^2 = 15^2 + 100^2 = 10225.$ Maybe you haven't covered such relationships yet. If not, then I suspect soon. $\endgroup$
    – BruceET
    Commented Apr 1, 2020 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.