# How to define the estimated standard error

I cannot find a proper definition of estimated standard error. Suppose $$X_1,\cdots,X_n\sim Bernoulli(p),$$ then $$\hat{p} = \dfrac{1}{n}\sum X_i$$ is the unbiased estimation of mean $$E[\hat{p}] = p.$$ And we know the standard error of $$\hat{p}$$ is $$se(\hat{p}) = \sqrt{p(1-p)}.$$ However the book said the estimated standard error of $$\hat{p}$$ is $$\hat{se}(\hat{p}) = \sqrt{\hat{p}(1-\hat{p})}.$$ How could we obtain the above equation? Just replace $$p$$ by $$\hat{p}$$ in the standard error? As $$\hat{p}$$ is unbiased?

The book is All of Statistics A Concise Course in Statistical Inference page 91.

• You don't state whether you have independence. If $\hat{p}$ is the maximum likelihood estimator for $p$ (in the usual situation it is) then that estimated standard error will be the ML estimator for the standard error. – Glen_b -Reinstate Monica Nov 2 '19 at 5:57

Actually there is never the estimator, but an estimator. But you can speak of the estimator, if it performs "good" from several statistical points of view, with no better alternative. Than it is some kind of textbook estimator.

This is often the case with Maximum Likelihood estimators, where the ML principle is one of the most succeeding principles in statistics.

In your case, the estimator $$\hat{p}$$ is an ML estimator for $$p$$.

The standard error is a function of $$p$$, lets call it $$g$$, i.e. $$g(p) = \frac{\sqrt{p(1-p)}}{\sqrt{n}}$$.

There is a property of ML estimators, which says, if $$\hat{\theta}$$ is an ML estimator of $$\theta$$, then $$f(\hat{\theta})$$ is an ML estimator for $$f(\theta)$$.

Therefore in your case $$g(\hat{p})=\frac{\sqrt{\hat{p}(1-\hat{p})}}{\sqrt{n}}$$ is an ML estimator for $$g(p) = \frac{\sqrt{p(1-p)}}{\sqrt{n}}$$.

Because it is an ML estimator, the textbook estimator for this problem, it is the estimator for the standard error.

• Having trouble making sense of this: The term 'standard error' is usually used for 'standard deviation' of an estimator. So for $n=1$ Bernoulli observation $X,$ you have estimator $\hat p = X.$ Unbiased because $E(\hat p) = p.$ And $Var(\hat p) = p(1-p), SD(\hat p) = \sqrt{p(1-p)}$ as $SE(\hat p).$ // Then you seem to say $\widehat{SE}(\hat p) = \sqrt{X(1-X)},$ which is $\equiv 0$ and does not seem useful. // Could you state the objective of finding estimated standard error and in what context? – BruceET Nov 2 '19 at 20:50
• "// Then you seem to say ..." I don't know, how you come to that conclusion, could you please elaborate this. – ghlavin Nov 2 '19 at 21:09
• You say you are in the process of revising your question. I will reserve any further comments until you have a chance to do that to your satisfaction. – BruceET Nov 2 '19 at 21:13
• Now I divided the term in function g, as I said. – ghlavin Nov 2 '19 at 21:18

Normal data, t procedures. For $$X_1, \dots, X_n$$ from a normal population, it is straightforward to say that $$Var(\bar X) =\sigma^2/n$$ and hence that $$SD(\bar X) = \sigma/\sqrt{n}$$ is the standard error of the mean.

Then, because $$\sigma^2$$ is estimated by $$S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2,$$ we can say that $$S/\sqrt{n}$$ is the (estimated) standard error of the mean.

Also, the t test for $$H_0: \mu = \mu_0$$ vs $$H_a: \mu \ne \mu_0$$ uses the test statistic $$T = \frac{\bar X - \mu_0}{S/\sqrt{n}} \sim \mathsf{T}(\nu = n-1),$$ rejecting at the 5% level of significance for $$|T| > t^*,$$ where $$t^*$$ cuts 2.5% of the probability from each tail of that t distribution.

Furthermore, 'inverting the test', $$\bar X \pm t^*\frac{S}{\sqrt{n}}$$ is a 95% confidence interval for $$\mu.$$ Givwn observed data, one might say that the confidence interval is a collection of $$\mu_0$$'s for which $$H_0$$ is not rejected.

Normal approximations for binomial data. However, if $$\hat p = X/n,$$ where $$X \sim \mathsf{Binom}(n,p).$$ an approximate normal test of $$H_0: p = p_0$$ vs $$H_0: p \ne p_0,$$ uses the statistic

$$Z = \frac{\hat p - p_0}{\sqrt{p_0(1-p_0)/n}},$$

rejecting $$H_0$$ at the 5% level when $$|Z| \ge 1.96.$$ [As a rough rule, this approximate test works well for moderately large $$n$$ and $$p_0$$ not far from $$1/2;$$ in particular, $$\min(np_0, n(1-p_0))>10$$ and $$0.2 < p_0 < 0.8.]$$

By contrast with the t interval above, the approximate Wald "95%" confidence interval $$\hat p \pm 1.96\sqrt{\frac{\hat p(1-\hat p)}{n}}$$ is not an inversion of the test above.

Speaking very loosely, one might say that $$\sqrt{\frac{\hat p(1-\hat p)}{n}}$$ is an estimated standard error of $$\hat p,$$ but this involves a second approximation beyond the normal approximation of the test. So you are correct to question its use.

The result is that the Wald interval does not reliably give the promised 95% coverage probability when $$n$$ is of small or moderate size, and its use for such $$n$$ has been deprecated. [The Wald interval was derived as an asymptotic approximation, and is OK for large $$n.]$$

The Agresti-Coull CI $$\tilde p \pm 1.96\sqrt{\frac{\tilde p(1-\tilde p)}{n+4}},$$ in which $$p$$ is estimated by $$\tilde p = \frac{X+2}{n+4},$$ is a more accurate CI for $$p$$ because it very nearly inverts the test. [See Wikipedia on binomial confidence intervals for additional details. Foe more on interval estimates of binomial proportions see this Q&A.]