How to Find Confidence Level of Results

I am curious to find out the confidence level of results. It's best to give an example of what I mean cause I struggle to speak in the technical terms.

Example: Jack runs an advertisement that 1000 people see. 10 of those people convert and actually purchase something. Jack is sure his conversion rate is 1%.

Sally disagrees. She thinks Jack doesn't have enough of a statistical confidence level in this result. What is the confidence level of 1% being the result.

A formula to get that would be great.

EDIT: I'm trying to add a statistical element to my marketing for my business. I plan to run a lot of advertisements so am trying to find a way to understand how many results I need to collect before I can prove or disprove a hypothesis I am testing.

Let's say for example my hypothesis is that running a certain ad will result in a conversion rate of 3% of people clicking the ad overtime.

For now, let's say I have just started the experiment and out of 1000 people, 30 did click it. So it seems that the the conversion rate (or probability of a person click the ad) is 3% (0.03 in probability).

However, I shouldn't trust these results because the sample size is too small. Instead I want to know what is probability that 3% (+ or - a margin of error) is indeed the conversion rate?

I thought I'd have to use a confidence intervals but now I'm thinking I should probably something that involves variance. I'll do some more research and update this when I find more. If you have any thoughts in the mean time on how to solve this problem, please let me know. Much obliged.

• How about a numeric tool? Is this homework? Sep 6 '16 at 13:55
• A confidence level has to do with confidence intervals, not point estimates/point forecasts. You might have some measure of uncertainty (standard error, root-mean-square prediction error etc etc) about the estimate or forecast, or indeed, you might perhaps generate a confidence interval (among other possibilities), but you'll have to help us understand more about what kind of thing you're talking about. What did you mean by 'confidence level'? NB with continuous variables the probability that a point forecast is exactly correct will be zero Sep 6 '16 at 14:46
• My apologies for my incompetence. I'll give so more detail in the original question. Sep 6 '16 at 17:53
• @JackRobson How did you solve this if I may ask? Oct 26 '20 at 8:02
• "Instead I want to know what is probability that 3% (+ or - a margin of error) is indeed the conversion rate?" This sounds like you want a Bayesian posterior probability for an interval 3% + or - a margin of error. For this you need to specify a prior distribution and then run standard Bayesian calculus. Oct 26 '20 at 12:01

Since you are uncertain about what may be the true conversion rate of the ad, you should consider using bayesian approach:

Let $$X$$ be an r.v such that $$X=1$$ if the ad is successful. So $$X \sim Be(p)$$, where $$p$$ is the conversion rate of the ad and let's assume further that $$p \sim U(0,1)$$. So basically you have a no prior belief about how efficient the ad might be (this of course can be changed by changing the prior distribution).

Further for $$n$$ independent $$X_i$$, $$Y \equiv {\sum_n X_i} \sim Bin(n,p)$$

From Bayes' theorem:

$$f(p| Y=y) = \frac{Pr(Y=y|p)f(p)}{Pr(Y=y)}$$

From distribution of $$Y$$, we know that $$Pr(Y=y|p) = {n\choose y} p^y(1-p)^{n-y}$$ and $$f(p)=1$$

Unconditional probability of $$Y=y$$: \begin{align} Pr(Y=y)&={n\choose y}\int_0^1 p^y(1-p)^{n-y} f(p)dp \\ &= {n\choose y}\int_0^1 p^y(1-p)^{n-y}dp \\ &= {n\choose y}\frac{y!(n-y)!}{(n+1)!} \\ &= \frac{1}{n+1} \end{align}

See this for the proof of above integral.

So we have:

$$f(p| Y=y) = {n\choose y}(n+1)p^y(1-p)^{n-y}$$

So you now have a full posterior distribution of conversion rate. You can simulate this distribution and/or sample from it to construct your credible interval, preferably a highest posterior density interval, to get an estimate of upper and lower bounds of your estimate for a given level of confidence.

EDIT: Embarrassingly, didn't notice that the posterior is beta distribution.

So, $$f(p| Y=y) \sim Beta(\alpha, \beta)$$

where $$\alpha = y+1$$ and $$\beta=n-y+1$$

Therefore for credible interval at 5% significance can be calculated using R:

> n=1000
> y=10
> a=y+1; b=n-y+1
> qbeta(0.025,shape1 = a,shape2 = b) # p_lower
 0.005498084
> qbeta((1-0.025),shape1 = a, shape2 = b) # p_upper
 0.01829503

$$Pr(0.0055 \leq p \leq 0.0183 | Y=10)=0.95$$

The classical parametric approach in frequentist statistics would be to model the situation as follows. Suppose all the people who see a certain advertisement are all independent of each other and have the same probability $$p$$ of clicking on the advertisement. We say these people form a Bernoulli population, i.e. a member $$X$$ of this population either reacts ($$X=1$$) to the advertisement or does not ($$X=0$$) with probabilities $$p$$ and $$1-p$$ respectively. Suppose you now take a sample of size $$n$$ ($$n=1000$$ in your case) of the population, so you observe $$X_1, \ldots, X_n$$ and see how many of them have clicked. In other words, we observe the sample average $$\bar{X} = \frac{1}{n} \sum_{i=1}^n X_i$$ In your case, you observe $$\bar{X}=0.01$$ and you are using this sample average as an estimate of the unknown population parameter $$p$$, i.e. $$\hat{p}=\bar{X}$$.

The statistical exercise we have to make now is deciding how 'reliable' or how 'precise' this estimate is. The central question here is: assuming the population fraction is indeed $$p$$, then what values of our estimator $$\bar{X}$$ are we likely to observe in a sample of size $$n$$? This brings probabilities into to the story. Specifically, we know (possibly to be explained further) that the sample sum $$n\bar{X}$$ is binomially distributed: $$\text{Prob}[n\bar{X} = i] = \binom{n}{i} p^i(1-p)^{n-i}\,,\qquad i=0,\ldots,n$$ This is the point now where most people will suggest the approximation $$\bar{X}\sim \text{N}(p,\frac{p(1-p)}{n})$$ which works fine if $$n$$ is large compared to $$np$$. So instead of the discrete binomial distribution, we now use the continuous normal distribution which is somewhat easier to work with. For example, it follows (could also be explained further) that $$\frac{\bar{X} - p}{\sqrt{\frac{p(1-p)}{n}}} \sim \text{N}(0,1)$$ has the standard normal distribution. Note that this distribution now no longer depends on $$p$$ or any other population parameter and for that reason, we call this quantity a pivot. Pivots are extremely useful to construct hypothesis tests and/or confidence intervals from. In our case, let $$z_{\alpha/2}$$ and $$z_{1-\alpha/2}$$ be the left and right quantiles of the standard normal distribution (in case $$\alpha=0.05$$ then these are the familiar values -1.96 and 1.96 respectively), then using the known distribution of the pivot: $$1-\alpha = \text{Prob}[ z_{\alpha/2} < \frac{\bar{X} - p}{\sqrt{\frac{p(1-p)}{n}}} < z_{1-\alpha/2} ]$$ which after some manipulations becomes $$1-\alpha = \text{Prob}[ \bar{X} - z_{1-\alpha/2} \sqrt{\frac{p(1-p)}{n}} < p < \bar{X} - z_{\alpha/2} \sqrt{\frac{p(1-p)}{n}} ]$$ These bounds form an (unobserved) confidence interval. Note that the left and right bound are not proper statistics. That means that strictly speaking, these left and right bounds can not be computed from a sample because we don't know what $$p$$ is! Yet another pragmatic approximation is to use the estimate $$\hat{p}=\bar{X}$$ instead of $$p$$ in these bounds. That brings us finally to: $$\text{a } (1-\alpha)\text{-CI for }p\text{ is }[\bar{X} - z_{1-\alpha/2} \sqrt{\frac{\bar{X}(1-\bar{X})}{n}} , \bar{X} - z_{\alpha/2} \sqrt{\frac{\bar{X}(1-\bar{X})}{n}}]$$ This often used way of constructing a CI is the culmination of two approximations: First we use CLT to apprimate the binomial distribution by a normal distribution and then we estimate the standard error by using $$\hat{p}=\bar{X}$$. In most cases however, the resulting CI is useful. Sometimes however, the lower bound can be negative or the upper bound larger than 1 which makes no practical sense for a parameter like $$p$$.

For $$\alpha=0.05$$, $$n=1000$$ and $$\bar{X}=0.01$$ we have the CI $$[0.0038,0.0162]$$.

Assuming that all people make independent choices, we can model the experiment as draws $$X_i$$ from a Bernoulli($$p$$) distribution with $$E(X_i) = p$$ and $$\sigma^2(X_i) = p(1-p)$$. For large $$n$$ ($$n = 1000$$ in your case) the independent draws from the Bernoulli distribution can be modelled as a Normal distribution according to the Central Limit Theorem. We then have for our Maximum Likelihood Estimate $$\hat{p}$$ ($$\hat{p} = 1\%$$ in your first example and $$\hat{p} = 3\%$$ in your second one)

$$\frac{\hat{p} - p}{\sigma_p/\sqrt{n}} \sim N(0,1)$$

From this, we can construct the 'Confidence Interval'

$$\hat{p} \pm \frac{z_{\alpha/2}}{\sqrt{n}}\sigma(p)$$

where $$z_{\alpha/2}$$ is the required z-score for the interval to contain the true value with probability $$1-\alpha$$. In order to make this a true Confidence Interval we need an estimation for $$\sigma_p$$ and could for example use $$\sigma_p \approx \sigma_\hat{p} = \sqrt{\hat{p}(1-\hat{p})}$$.

In your first example we then obtain the 95% Confidence Interval [0.38, 1.62] (in percent) and in your second example [1.94 4.06].

Note that the second Confidence Interval is larger, despite the same number of people that participated in the experiment. This is due to the fact that $$\sigma_\hat{p} = \sqrt{\hat{p}(1-\hat{p})}$$ has its maximum for $$\hat{p} = 0.5$$. This fact is often used in opinion polls (for example for elections) that can be modelled as draws from a Bernoulli distribution as well. You will notice that the number of people that are being asked is often at around 1000, which leads to a margin of error $$\pm 3\%$$ at the 95% confidence level using the same calculation as above (under the assumption that $$p=0.5$$).