Why do "most" (basically all) statistics text books use $\hat{\sigma}^2=\hat{p} (1-\hat{p})$ as an estimator for the variance of a Bernoulli process which we know is biased. Should the default not be the bias corrected sample version $s^2 = \hat{p} (1-\hat{p}) \cdot n/(n-1)$, which is the default for non-binary data.
$\begingroup$
$\endgroup$
3
-
3$\begingroup$ Bias is not the ultimate property to seek. For instance, $s^2$ has a larger variance than $\hat\sigma^2$. (And all Bayes estimators are biased.) $\endgroup$– Xi'anCommented Jan 13, 2021 at 12:56
-
$\begingroup$ While true, your answer applies to all estimators and does not address my specific question why for binary data alone we seem to prefer the biased variance ("1/n") when for all other cases, the default sample variance is chosen to be the unbiased version ("1/(n-1)"). $\endgroup$– Markus LoecherCommented Jan 15, 2021 at 10:06
-
1$\begingroup$ I disagree with the closing of my question. None of the previous discussions addresses my very specific/narrow question as rephrased above. I am aware of the general scientific discourse on unbiased estimators which skirts the difference in textbook defaults for Bernoulli versus "non Bernoulli" sample variance. $\endgroup$– Markus LoecherCommented Jan 15, 2021 at 10:11
Add a comment
|