For 1000 newsletter recipients, I crudely estimate the likelihood (p) of them reading the next email sent as:

p = number_emails_read / number_emails_received

I also calculate the standard deviation for each recipient. But there's a problem..

The problem

Anyone who received a small number of emails, say 2 emails, and read both, has 100% estimated likelihood to read the next one, and standard deviation of 0 (in other words, the standard deviation tells us the estimate of 100% likelihood is extremely accurate).

But in reality, their likelihood may actually be much lower than 100%, say 50%, and they just happened to read the two sent to them, but may not be anywhere close to 100% likely to read the next email.

In such cases, the very small sample size may lead us to a false positive, and given the inconvenience of receiving junk emails, I want to create a bias against such cases, to reduce false positives (probably at the expense of false negatives, but that is acceptable, if not desirable here)


How can we adjust against this, that is, to penalise the small sample sizes so that we avoid false positives?

What I know so far

A very crude solution could be to simply remove all cases where the sample size is less than a certain value (e.g. <10), so as to avoid the highest risks of false positives.

But I hope there is a more sensible / statistically valid solution


One standard way to do this is to give an interval estimate of $p$ instead of just a point estimate. Here is a very brief introduction.

I think you are assuming that a recipient makes a separate independent decision whether to answer each email.

Wald confidence interval. If a recipient answers $x =8$ out of $n=10$ emails, then you might say that the point estimate is $\hat p = x/n = 0.8.$ Then one kind of interval estimate is a Wald 95% confidence interval of the form $\hat p \pm 1.96\sqrt{\frac{\hat p(1-\hat p)}{n}},$ which amounts to the interval $(0.77, 0.83).$ Roughly speaking, this says that over the long run the recipient may answer between 77% and 83% of emails.

[This interval is based on a normal approximation to the binomial distribution. The numbers $\pm 1.96$ include 95% of the probability in a standard normal distribution. This kind of interval works well for very large $n,$ but not so well for small $n.]$

Agresti-Coull interval. A slight adjustment gives point estimates and confidence intervals that work better for small $n$ and just as well for large $n:$ The idea is to use the point estimate $\tilde p = \frac{x+2}{n+4}$ and the so-called Agresti-Coull 95% confidence interval $\tilde p \pm 1.96\sqrt{\frac{\tilde p(1-\tilde p)}{n+4}}.$ For 8 answers out of 10, this computes to point estimate $\tilde p = 10/14 = 0.714$ and 95% confidence interval $(0.69, 0.74).$

Larger numbers of emails. Now lets look at the point and Agresti-Coull interval estimate for someone who gets more email. Suppose this recipient answers $x = 80$ out of $n=100$ emails. Then $\tilde p = 82/104 = 0.79$ is the point estimate and the 95% interval estimate is the much narrower $(0.785. 0.792).$ More data gets you a 'more precise' interval.

Notes: (1) At the start, don't try to figure out the reason for the inflated numerator and denominator used in the Agresti-Coull method. [There is a sound theoretical basis, but the last step of that is a simplifying trick.]

(2) Confidence intervals won't work so well for people at the extremes, say $x = 0$ or $2$ answers out of $n = 2$ emails. Trying to summarize overall behavior with such small amounts of data doesn't make much sense. Also, you may get puzzling results for $0. 1, 99,$ or $100$ answers out of $100$ emails. (Maybe such a person isn't making $100$ independent decisions, acting out of quirky habits that are subject to change.)

(3) If you want more explanation, you can read technical details about confidence intervals in an elementary statistics text or on Wikipedia, which shows several styles of confidence intervals that I didn't mention (some of them messy). There's also a Youtube demo, which I confess I didn't watch.

  • $\begingroup$ Fantastic answer, thanks so much. In my case about 80% of the dataset is actually small sample size (<7) - is there anything more to add fo small samples? Or is the conclusion just that there is only so much information we can glean from a small sample? $\endgroup$ – stevec Apr 5 '20 at 8:01
  • 1
    $\begingroup$ Then maybe consider a Jeffreys probability interval, which is a Bayesian interval based on a noninformative prior distribution. Suppose you have 2 responses out of 7. In R, the statementx = 2; n = 7; qbeta(c(.025,.975), x+.5, n-x+.5) returns interval endpoints 0.06472827 and 0.64766165, which you might round to 2 or 3 places. Also x=20, n=40 gives interval (0.1901208, 0.3986144). $\endgroup$ – BruceET Apr 5 '20 at 8:08
  • 1
    $\begingroup$ Opps! Typo at end of last comment: n = 70 (not n = 40). Even if you're not used to doing Bayesian analysis, the Jeffreys intervals work well and can be interpreted as ordinary confidence intervals. $\endgroup$ – BruceET Apr 5 '20 at 8:26
  • 2
    $\begingroup$ In R binom.test(read, received)$conf.int can be used. If the number read equals the number received and n is the common value then the confidence interval will be approximately [1-3/n, 1] for sufficiently large n. This is known as the rule of 3. en.wikipedia.org/wiki/Rule_of_three_(statistics) Also see binom.confint in the R binom package. That function implements several methods and the exact method gives the same result as binom.test. $\endgroup$ – G. Grothendieck Apr 5 '20 at 10:58
  • $\begingroup$ @G.Grothendieck. Thanks for useful details about R. $\endgroup$ – BruceET Apr 5 '20 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.