How to bias against false positives given estimate and standard deviation?

Question

For 1000 newsletter recipients, I crudely estimate the likelihood (p) of them reading the next email sent as:

p = number_emails_read / number_emails_received

I also calculate the standard deviation for each recipient. But there's a problem..

The problem

Anyone who received a small number of emails, say 2 emails, and read both, has 100% estimated likelihood to read the next one, and standard deviation of 0 (in other words, the standard deviation tells us the estimate of 100% likelihood is extremely accurate).

But in reality, their likelihood may actually be much lower than 100%, say 50%, and they just happened to read the two sent to them, but may not be anywhere close to 100% likely to read the next email.

In such cases, the very small sample size may lead us to a false positive, and given the inconvenience of receiving junk emails, I want to create a bias against such cases, to reduce false positives (probably at the expense of false negatives, but that is acceptable, if not desirable here)

Question

How can we adjust against this, that is, to penalise the small sample sizes so that we avoid false positives?

What I know so far

A very crude solution could be to simply remove all cases where the sample size is less than a certain value (e.g. <10), so as to avoid the highest risks of false positives.

But I hope there is a more sensible / statistically valid solution

Answer 1

One standard way to do this is to give an interval estimate of $p$ instead of just a point estimate. Here is a very brief introduction.

I think you are assuming that a recipient makes a separate independent decision whether to answer each email.

Wald confidence interval. If a recipient answers $x =8$ out of $n=10$ emails, then you might say that the point estimate is $\hat p = x/n = 0.8.$ Then one kind of interval estimate is a Wald 95% confidence interval of the form $\hat p \pm 1.96\sqrt{\frac{\hat p(1-\hat p)}{n}},$ which amounts to the interval $(0.77, 0.83).$ Roughly speaking, this says that over the long run the recipient may answer between 77% and 83% of emails.

[This interval is based on a normal approximation to the binomial distribution. The numbers $\pm 1.96$ include 95% of the probability in a standard normal distribution. This kind of interval works well for very large $n,$ but not so well for small $n.]$

Agresti-Coull interval. A slight adjustment gives point estimates and confidence intervals that work better for small $n$ and just as well for large $n:$ The idea is to use the point estimate $\tilde p = \frac{x+2}{n+4}$ and the so-called Agresti-Coull 95% confidence interval $\tilde p \pm 1.96\sqrt{\frac{\tilde p(1-\tilde p)}{n+4}}.$ For 8 answers out of 10, this computes to point estimate $\tilde p = 10/14 = 0.714$ and 95% confidence interval $(0.69, 0.74).$

Larger numbers of emails. Now lets look at the point and Agresti-Coull interval estimate for someone who gets more email. Suppose this recipient answers $x = 80$ out of $n=100$ emails. Then $\tilde p = 82/104 = 0.79$ is the point estimate and the 95% interval estimate is the much narrower $(0.785. 0.792).$ More data gets you a 'more precise' interval.

Notes: (1) At the start, don't try to figure out the reason for the inflated numerator and denominator used in the Agresti-Coull method. [There is a sound theoretical basis, but the last step of that is a simplifying trick.]

(2) Confidence intervals won't work so well for people at the extremes, say $x = 0$ or $2$ answers out of $n = 2$ emails. Trying to summarize overall behavior with such small amounts of data doesn't make much sense. Also, you may get puzzling results for $0. 1, 99,$ or $100$ answers out of $100$ emails. (Maybe such a person isn't making $100$ independent decisions, acting out of quirky habits that are subject to change.)

(3) If you want more explanation, you can read technical details about confidence intervals in an elementary statistics text or on Wikipedia, which shows several styles of confidence intervals that I didn't mention (some of them messy). There's also a Youtube demo, which I confess I didn't watch.