Confidence / statistical significance of empirical probability estimation

Question

The empirical probability of an event can be estimated as the number of times the event occurred $E$ divided by the total number of samples $N$.

For example rolling a dice 12 times if we see a 6 for 3 times we can say that $\hat p =P(x=6) = 3/12$.

How can I add in this model the confidence that I have in the estimated probability as a function of the samples size?

For example if $N = \infty$ then the confidence should be $1$ while if $N\sim 0$ the confidence should be 0 .

The main application I am looking for is to not overestimate the probability of certain events that may only be due to the small sample size. For example if you roll a dice twice and you get 2 times 6 you do now want to say that the probability of 6 is 100%.

A current simple solution that I am using is $\hat p = E / (N + K)$ where $K$ is predefined value. I am wondering if a better solution exists.

Answer 1

The problem is not as easy as it seems. When rolling a dice or flipping a coin:

• the parameter is $p$: theoretical probability to get a 6
• you known $X$: the number of 6s on $N$ trials
• you want to estimate $p$.

The distribution of $X$ is a binomial distribution $B(N,p)$. What is the best estimator for $p$?

The notion of the best estimator is easier in the Bayesian approach: the estimator with least expected squared error is the conditional mean $\hat p=E(p|X)$ . With a uniform prior it is $\hat p=\frac{X+1}{N+2}$. See: https://en.wikipedia.org/wiki/Checking_whether_a_coin_is_fair. You can calculate this for the prior that you want. With a Beta prior with parameters $\alpha,\beta$ it is $\hat p=\frac{X+\alpha}{N+\alpha+\beta}$, which is equivalent to the frequentist estimator having seen $\alpha$ 6s on $\alpha+\beta$ trials before. Your simple solution is the special case $\alpha=0$.

In the frequentist approach, the notion of the best estimator is nothing simple. The estimator used is usually $\hat p=\frac{X}{N}$. You can compute confidence intervals. See https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

Now, from a machine learning point of view, I recommend analysing the sentence:

The main application I am looking for is to not overestimate the probability of certain events that may only be due to the small sample size.

How do you quantify the consequences of overestimation? This kind of analysis leads to find the best estimator for your specific purpose. You can look at it for example: https://en.wikipedia.org/wiki/Multi-armed_bandit.

Answer 2

If I understand correctly you have a binomial distribution of $N$ trials and you estimate the success probability $p$ using $\hat{p} = E/N$.

As covered by the answers to this question you can derive a standard error for $\hat{p}$ equal to $\sqrt{\frac{\hat{p}(1-\hat{p})}{N}}$.