3
$\begingroup$

I'm trying to self learn some topics in Statistics by reading Statistics by David Freedman.

There is something that I cannot understand regarding the standard error. I have been reading in Google that the standard error is a little vague, and we should indicate what estimator we are evaluating (i.e., SE of the mean for example), which decreases as we have more samples (and it makes sense to me).

Now, Freedman is using the SE term to define what, for me, is the SD of the sum samples, see point 2 here (pdf) with the formula he used in the book.

2. The Standard Error for a SUM (17.2)
The standard error is a measure of the chance error. An outcome (sum) from some number of draws will be around an expected value but it can (and will be) off by chance error.
Formula: standard error of a sum = square root (number of draws) times the standard deviation (SD) of the box. So for a coin box, in 10 tosses, the standard error = 10 * 0.5 = 1.6 (approximately)

So is this formula in fact the SE or just the prediction of the SD?

$\endgroup$
  • $\begingroup$ The standard error of the sum is just an example of a standard error. $\endgroup$ – Michael M Jan 2 '19 at 20:31
  • $\begingroup$ the terms are basically interchangeable, the differences would be dependent on the field $\endgroup$ – Aksakal Jan 2 '19 at 20:46
3
$\begingroup$

Let the sum of the Heads in a binomial experiment (sequence of independent Bernoulli trials) be $X \sim \mathsf{Binom}(n = 16, p = 1/2).$ Then $E(X) = np, Var(X) = np(1-p),$ and $SD(X) = \sqrt{Var(X)} = \sqrt{np(1-p)}.$ If $p = 1/2,$ then $SD(X) = \frac 1 2\sqrt{n}.$

To an extent, the terminology 'standard deviation' and 'standard error' can be used interchangeably when referring to a random variable, especially if the random variable is viewed as an estimate of something.

Similarly, the random variable $\hat p = X/n$ is an estimate of $p.$ We have $E(\hat p) = p,$ $Var(\hat p) = \frac 1 {n^2}Var(X) = \frac{p(1-p)}{n},$ and $SD(\hat p) = \sqrt{\frac{p(1-p)}{n}}.$ If $p = \frac 1 2,$ then $SD(\hat p) = \sqrt{\frac{1}{4n}}.$

One possible point of confusion arises when $p$ is unknown and we use $\hat p$ to estimate $p.$ Then an estimate of $SD(\hat p)$ is $\sqrt{\frac{\hat p(1-\hat p)}{n}},$ So that the estimated standard error of $\hat p$ is $\sqrt{\frac{\hat p(1-\hat p)}{n}}.$ Perhaps because this is done so often, one sometimes drops the word 'estimated' and calls $\sqrt{\frac{\hat p(1-\hat p)}{n}}$ the standard error of $\hat p.$ Unfortunately, this common abuse of terminology is not always clarified in elementary textbooks.

$\endgroup$
  • $\begingroup$ Thank you - Just to confirm I got it, and can relate the SD with the SE. Suppose I have an excel an put random values (1 to 4) in 10 cells. This is my sample. Then: <br/> 1) Calculate the SD of those 10 numbers <br/> 2) Calculate the SE of the mean using the formula SD / SQRT(10) <br/><br/> And if I create, say N more samples of 10 random numbers, and for each sample I calculate the average, and calculate the SD of those averages, that SD should be the same as the previous SE? I tried that in excel and was quite similar, not sure if was by chance. $\endgroup$ – F_M Jan 13 '19 at 0:22
2
$\begingroup$

The standard error generally just refers to the standard deviation of an estimator (at least for an unbiased estimator). Hence, it makes no sense to refer to standard error without specifying what estimator you are talking about. Since the standard error depends on the behaviour of the estimator, through the behaviour of the sample values, it often depends on the model parameters in the statistical model. Hence, it is common to draw a distinction between the true standard error (which depends on model parameters), and the estimated standard error that comes from substituting the parameter estimates into the equation for the true standard error (which is a statistic).

Example: Suppose I have a sample $X_1,...,X_n \sim \text{Bern}(\theta)$ with unknown parameter $\theta$. In this case, the sample mean $\bar{X}$ is an unbiased estimator for $\theta$, and this estimator has true standard error:

$$\text{se}(\bar{X}) = \mathbb{S}(\bar{X}) = \sqrt{\frac{\theta (1-\theta)}{n}}.$$

You can see that the true standard error depends on the unknown parameter $\theta$, and so this is also unknown. We can substitute our estimator $\bar{X}$ for $\theta$ to get the estimated standard error:

$$\hat{\text{se}}(\bar{X}) = \sqrt{\frac{\bar{X} (1-\bar{X})}{n}}.$$

The estimated standard error is a statistic --- it is the thing that is generally reported in model output, such as in linear regression output, etc. (Some students find this latter concept a bit hard to grasp, since it is an estimator of the standard deviation of an estimator of another quantity.)

So far as I am aware, this is the proper way to use the term. Unfortunately, some writers do not always draw a clear distinction between the true standard error and the estimated standard error, and this sometimes leads to confusion on the topic. To make matters more confusing, when writers just say "standard error", and don't specify which of these two they are talking about, sometimes they are referring to the true standard error, and sometimes they are referring to the estimated standard error. Nonetheless, you can generally tell which is the case by looking at the context and reading the relevant formulae to see if they have parameters in them. If the "standard error" is an expression depending on the model parameters then it is the true standard error, and if the "standard error" is an expression depending only on the data (i.e., if it is a statistic) then it is the estimated standard error.

$\endgroup$
2
$\begingroup$

In statistics, you generally distinguish between population and sample. The population is the set of all possible measurements, that can ever be taken. The population is therefore closely related to a random variable. A sample, on the other hand, refers to some actual measurements collected from the population. So for a 6 sided dice, the population would be the infinite number of rolls ever to be performed. A sample is when you roll the dice $N$ times.

Standard deviation (as well as expected value) is a description of the population (or the random variable). How much can you expect the individual measurements from the population to be spread. True standard deviation can only be calculated if the underlying distribution of the population is known (it usually is not). From a sample, the standard deviation can only be estimated.

Standard error is a measure describing all possible samples (so basically the population of samples). Assume the six sided dice again, and you do 10 rolls and get the mean. It should vary around three, but each different sample of 10 rolls will give you a different mean. So the mean here is an estimator of the expected value. How much it varies is the standard error of the mean. Basically, it is asking "what would be the error on average, if I use this estimator". Similarly, if you just add the 10 rolls instead of taking the mean, you get an estimator of the sum of 10 arbitrary rolls. The expected value would be around 30. However, each sample of 10 rolls will give you a different estimate. The error of the estimator (based on each sample) is the standard error of the sum.

So standard deviation is a description of a population, standard error is a description of an estimator (which is actually a random variable, so it also has a population). Basically the standard error is the standard deviation of the estimator. Finally, since the standard error describes a random variable, it can only be estimated based on a sample, so the usual formulas to get the standard error from a sample are actually calculating another estimator for the true standard error.

$\endgroup$
  • $\begingroup$ in this case, the "estimator" would be the mean, for example? In other words, if you have subsets of your data and calculate the mean for each, the standard error would be the standard deviation of these subset means? $\endgroup$ – Tilen Jan 5 '20 at 11:34
  • $\begingroup$ @Tilen Yes, the mean is an estimator of the expected value. You are confusing population and sample somewhat. Population is the set of all possible observations you could ever make (e.g. all possible datasets you could have gathered). A sample is a realization from the population (e.g. a dataset). Yes, you could estimate the standard error by creating subsets of the dataset. But that is something different. This would be resampling to get a more reliable estimator of the standard error. The standard error still is a description of the population, not the sample. $\endgroup$ – LiKao Jan 7 '20 at 10:51
1
$\begingroup$

The Standard Deviation is a term usually reserved to indicate the square root of the average squared distance between each observation and its mean. The Standard Error typically means the square root of the average squared distance between an estimated value (i.e. an estimated parameter like $\hat{\beta}$) and its true population mean.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.