# Analytic or sample standard deviation with binomial data

I've been looking for recommendations on whether it's better to use the sample standard deviation (SD) for a binomial distribution or use the analytic SD (or variance). It's for experiments with accuracy data so each subject contributes several binomial (correct or incorrect) responses resulting in an accuracy score between 0 an 1 (Ncorrect / Ntotal). I get the mean across subjects. Assuming each one contributes the same amount should I used the sample SD of the individual accuracy scores or use the analytic equation (sqrt(pq / n) where n is the number of responses / subject)?

This is a query for descriptive purposes only. I could use a multi-level logistic regression to model but I'm just looking for simple descriptives.

Example: Each subject gets 20 tries at the task and there are 6 subjects with accuracy 0.95, 0.80, 0.80, 0.65, 0.90, 0.70. The mean accuracy would be 0.8 and therefore the analytic SD of that accuracy is sqrt(0.8 * 0.2 / 20) = 0.089. However, the SD of those six numbers calculated on the sample is 0.114. Which is the better SD estimate to use?

• Could you show us a simple example where the "analytic" and "sample" variance formulas actually give different numbers? That would help us appreciate what you are trying to ask.
– whuber
Commented Aug 21, 2014 at 19:43
• I put in an example but they almost always give different numbers with large enough numbers of test items (i.e. fine enough measurement resolution). They may tend to come out close, but not the same.
– John
Commented Aug 21, 2014 at 20:32
• I think I understand the question. But I have no idea of a possible motivation to report an estimate of the variance of a binomial distribution. Commented Aug 21, 2014 at 21:16
• Stéphane, because reporting the central tendency and variability of a sample is a good thing to do? I changed variance to SD because it's likely that future people looking up a similar question might be more interested in the SD report.
– John
Commented Aug 21, 2014 at 22:16
• @Stéphane Wouldn't an appropriate comparison of the sample SD to a binomial SD estimate provide information about over- or under-dispersion?
– whuber
Commented Aug 21, 2014 at 22:28

"Better" depends on context and purpose. Before addressing this issue, though, let's consider the data.

As a point of departure we might assume--hypothetically, being willing and happy to be proven wrong later in the analysis--that the outcomes of each subject's attempt at the task are independent. This permits us to hold up a simple model for scrutiny, one in which each subject $i$ has a constant chance $p_i$ of success with each attempt. It follows that the raw counts of successes $(x_i,\ i=1, 2, \ldots, n)$ consist of six (or, more generally, $n$) independent realizations of Binomial$(m, p_i)$ variables $X_i$ (with $m=20$ in this case). In this case the raw counts are $(19,16,16,13,18,14)$, obtained by multiplying the reported success rates by $20$.

This is a complicated model because it has as many parameters ($n$ of them) as there are data. To see whether the complication is worthwhile, we ought to compare this model to a simplified version. The simplest is that all the $p_i$ are equal: the subjects have equivalent abilities at the task. Is there a small set of simple, easily understood, summary statistics that might help give us some quick insight into which model would be appropriate?

In the spirit of an Analysis of Variance we might be inclined to compare the variance of the dataset--which will comprise the variances inherent in each of the $X_i$ together with the variance of the $p_i$--to some measure of the variance to be expected when all the $p_i$ are equal. Therefore we compute:

1. The mean of the $p_i$ is $p = (1/n)\sum_{i=1}^n p_i.$ There are several ways to estimate this, but one of the simplest--as justified by the hypothesis that all the $p_i$ are equal--is the sample mean,

$$\hat{p} = \frac{1}{n}\sum_{i=1}^n \frac{x_i}{m} = \frac{4}{5} = 0.8.$$

2. The variance of each $X_i$ is $m p_i(1-p_i)$; under the hypothesis of equality, this is $m p(1-p)$, which can be estimated as

$$\hat{\sigma} = m \hat{p} (1 - \hat{p}) = \frac{16}{5} = 3.2.$$

3. The variance of the data is

$$\text{Var}(x_i) = \frac{1}{n}\sum_{i=1}^n (x_i - m \hat{p})^2 = \frac{13}{3} = 4.\bar{3}.$$

Please notice that, in the spirit of description and exploration, division by $n-1$ in this variance calculation could be considered irrelevant. However, should one feel a need to so change the denominator, the result would be $26/5 = 5.2$.

The statistic (3) can be understood as arising from two components: the variation in subject performances due to chance plus the variation in capabilities between the subjects. That is why the two standard deviations computed in the question (which are the square roots of (2) and (3)) may differ. It becomes clear that they work together to give two separate pieces of information about the data.

It is attractive to take one more step. ANOVA teaches us that the relevant statistic to examine would be the ratio

$$\text{Var}(x_i) / \hat{\sigma}.$$

A value much greater than $1$ would indicate the $p_i$ should be treated as non-constant. In the present case--using the alternative expression for the variance employed in the question--this ratio equals $(26/5)/(16/5) = 13/8 = 1.625.$ This is precisely the square of the ratio of standard deviations reported in the question, $(0.1140/0.08944)^2$.

This analysis has provided a perspective in which the distinction between the two standard deviation calculations in the question can be both understood and used to gain insight into the data-generation mechanism:

1. The two calculations differ due to possible fluctuations in the subjects' capabilities.

2. Their ratio (when squared) can be interpreted as an ANOVA F-statistic, permitting its use in evaluating whether the apparent fluctuations may be due to chance or should be accepted as real.

To answer the question, then, one might wish to report both standard deviation calculations together with the F-like statistic given by the square of their ratio.

I've been looking for recommendations on whether it's better to use the sample standard deviation (SD) for a binomial distribution or use the analytic SD (or variance).

If it was an iid sample actually from a binomial distribution ... the MLE would be the usual analytic one; as such, at least in large samples, that's going to be a very good choice.

The problem is, the claim of being binomial isn't always as true as people might hope.

Summing 0-1 variables is not of itself sufficient to establish that something is binomial, since the binomial relies on independence and constant $p$, and frequently one or the other or both don't quite hold.

Do you know for certain that $p$ is constant, for example?

This is a query for descriptive purposes only. I could use a multi-level logistic regression to model but I'm just looking for simple descriptives.

If you're fitting a multilevel logistic model, that would suggest that $p$ isn't constant across all cases, in which case you shouldn't use the binomial to work out the standard deviation, and such a model would also suggest that independence (unconditionally) could be suspect.

If there's any doubt on constant $p$, make sure to quote the sample SD. If there's doubt on independence, it may be that neither measure is very meaningful, though it depends on what you're trying to achieve by quoting it.

Example: Each subject gets 20 tries at the task and there are 6 subjects with accuracy 0.95, 0.80, 0.80, 0.65, 0.90, 0.70. The mean accuracy would be 0.8 and therefore the analytic SD of that accuracy is sqrt(0.8 * 0.2 / 20) = 0.089. However, the SD of those six numbers calculated on the sample is 0.114. Which is the better SD estimate to use?

In these circumstances, I'd suggest you quote the sample SD (within-subject you might quote the analytic one, but beware of the possible effect of dependence within subjects). [The overall variance might be expected to be close to the sum of the within- and between- components.]

• Some good points regarding the modelling, I didn't mean there'd be an "and" I meant a different situation. Regarding the sample SD within subjects, there is no sample SD within subjects. You can only get an analytic one since each subject provides a single accuracy measure. Or perhaps I'm reading you wrong.
– John
Commented Aug 22, 2014 at 2:00
• I mentioned the analytic one as something you can quote within-subject on the assumption that's all you can do - if you had several such samples per subject, you'd again want to use the sample one. If comparing sd's calculated each way, though, you should beware - the "analytic" one actually corresponds directly to computing the sample sd on the original 0's and 1's without Bessel's correction (e.g. if you have a sample proportion of 5/9 you can write it as a sample of 5 $1$'s and 4 $0$'s and compute the s.d. with the n-divisor, you have the same result). ...(ctd) Commented Aug 22, 2014 at 2:46
• (ctd)... If you want to compare with a sample sd across several subjects, the proper comparison would be with the uncorrected (n-divisor) sd, since they're all on the same footing then. Commented Aug 22, 2014 at 2:47

reporting the central tendency and variability of a sample is a good thing to do

You report the central tendency with the mean accuracy. The question is, what should you use for the variability?

Note that your "analytical SD" only depends on the mean accuracy (and the number of tests) -- it thus can't give more information than the mean accuracy alone. Indeed, the analytical SD answers this question: "Assuming all subjects perform the same way, how big will the expected spread in the data be?" (There still will be some spread due to randomness.) I don't think that is what you want to report.

You want to report how much spread there actually is in your data. This is what your "sample SD" calculates, so you should report that.

N.B: You might want to compare your "sample SD" (how big is the actual spread) to your "analytical SD" (how big do we expect the spread if all subjects perform the same), to check whether some subjects performed better than others. But to do this properly, you should perform a significance test.

• I downvoted because your last sentence is not sensible. Even if they all clearly come from the same distribution some will pass a significance test for any reasonably sized experiment. With low numbers of binomial events per subject only very extreme deviations will pass a test, even if there really is strong heterogeneity. Also, a test doesn't tell you they're similar, which is what you want to know. And there would be huge multiple testing issues. The comparison might be a useful thing to do but a significance test is not.
– John
Commented Nov 6, 2017 at 4:34