What is the proper way to estimate the probability (proportion of time) a rare event occurs?

Question

Often, I need to estimate the probability (proportion of time) a rare event occurs. The standard MLE estimate often gives me extreme estimates since the denominator is usually 1, and the numerator is either 0 or 1, giving me either 100% or 0%.

For example, I am trying to estimate the proportion of web referrals as a result of my email campaign for each of my users. Since the events are rare, most of my users usually have only 1 web referral, and they have either 0 email referral or 1 email referral. In such cases, the MLE estimate is quite unreliable.

Are there standard tricks to correct this over-under estimation? Perhaps something like the laplace smoothing? If yes, how should I go about it?

Answer 1

Your question is close to this one about constructing confidence limits when the binomial estimate is either zero or one. But you have multiple users, and want to estimate a small probability for each one.

Then, if you are willing to assume this small probabilities are similar (your users are exchangeable in this regard), you can get strength from using all the data at once. In some way you could estimate one common $p$, and then the individual estimates could be shrinked towards this common estimate. That could be done in a Bayesian or empirical Bayesian way. This is also called a hierarchical bayes model.

Or, since you have many users, estimate the distribution of probabilities over the users! That leads you into mixed-models.