3
$\begingroup$

Why do "most" (basically all) statistics text books use $\hat{\sigma}^2=\hat{p} (1-\hat{p})$ as an estimator for the variance of a Bernoulli process which we know is biased. Should the first choice not be the bias corrected sample version $s^2 = \hat{p} (1-\hat{p}) \cdot n/(n-1)$, which is the default for non-binary data.

I am aware of the multiple related threads that seem similar to my question. However, I am not seeking a discussion on the merits of unbiased estimators or the origin/relevance of the $n/(n-1)$ correction.

My specific question is about teaching and communicating, namely why for binary data alone we seem to prefer the biased variance ($1/n$) when for all other cases, the default sample variance is chosen to be the unbiased version ($1/(n-1)$)

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ could you give at least one examples of the textbooks? $\endgroup$
    – seanv507
    Commented Jan 17, 2021 at 12:38
  • $\begingroup$ In most other cases, the test statistics based on the uncorrected estimator would follow distributions that are more complicated or onerous to tabulate, whereas that's not an issue in the Bernoulli case. Consideration of the tests for OLS, for instance, shows how one set of tables of t statistics will suffice provided the error variance is corrected by dividing by $n-p$ rather than $n$ itself. $\endgroup$
    – whuber ♦
    Commented Jan 17, 2021 at 18:46
  • 1
    $\begingroup$ @whuber Thank you for this VERY helpful and wise inferential perspective which would explain the limited upsides. I still wonder what the downside of using the unbiased version would be. Is it simply that p*(1-p) is so concise and elegant compared to p*(1-p)*n/(n-1)? $\endgroup$
    – Markus Loecher
    Commented Jan 19, 2021 at 10:18
  • $\begingroup$ I think it might depend on the application, Markus. I would keep an eye out for statistical procedures that appear to involve factors that are functions of $(n-1)/n,$ because that would suggest they could be simplified by using the unbiased estimator. But unless the target of estimation is $p(1-p)$ itself, it is hard to see how adjusting for the bias would be relevant to most statistical procedures. In particular, you cannot simultaneously estimate $p$ and $p(1-p)$ in an unbiased way. $\endgroup$
    – whuber ♦
    Commented Jan 19, 2021 at 14:04
  • $\begingroup$ Thanks again, especially for the reminder that Bernoulli/Binomial are one-parameter distributions, which I agree is relevant for my question. $\endgroup$
    – Markus Loecher
    Commented Jan 21, 2021 at 22:37

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.