How to estimate the probability of a binary outcome and generalize to a population? Let me give you the real life goal I am trying to accomplish:
I have database A and I want to add a value from another database, database B, to A. The best common value between them to match on is a phone number. However this is not guaranteed to be unique and occasionally a value from B will be mapped to A falsely.  
I would like an estimate of the likelihood than a record will have a wrong value mapped to it from B.
Verifying the correct mapping needs to be done by hand so it is impractiacl to check every record in the data set. A sample of some size needs to be used. 
I want to know the best technique, procedure, or distribution to use to get an estimate of the likelihood that a record is incorrectly mapped and a confidence interval around that estimate when generalized to the whole data set. 
I have looked into binomial distributions, but I am not sure what to do when the probability is unknown and is not determinant like a coin-toss(the common example) 
Regards
 A: If you are willing to assume that it is a reasonable approximation of the truth that the probability of an incorrect match follows a Bernoulli distribution independent of other matches then the following technique can be used to obtain an estimate and 95% confidence interval of this probability (a Bernoulli distribution is a binomial distribution with just 1 trial).
Suppose we sample $n$ database pairings, and let $x_i$ have value 0 if the pairing is correct, and 1 if the pairing is incorrect.  Let $\pi$ denote the true probability that a pairing is incorrect.  Then it can be shown that the maximum likelihood estimator (i.e. one choice of reasonable estimator) is given by
$$ \hat{\pi} = \frac{1}{n} \sum_{i=1}^n x_i $$
where $\hat{\pi}$ denotes this estimator and $x_i$ denotes whether the $i^{\text{th}}$ pairing was incorrect ($x_i = 1$) or correct ($x_i = 0$).  Then it follows from the Central Limit Theorem that
$$ \frac{ \sqrt{n} \left( \hat{\pi} - \pi \right) }{ \sigma } \stackrel{L}{\rightarrow} N(0,1 ) $$
where $\sigma^2$ denotes the variance of the Bernoulli random variable (and hence $\sigma$ is the square root of the variance, and is known as the standard deviation).  The side to the right of the arrow denotes a normally distributed random variable with mean 0 and variance 1 (often called a standard normal random variable).  The arrow means that as the sample size $n$ gets larger and larger that the distribution on the left becomes arbitrarily close to the distribution on the right.
Next, it can be shown that for a Bernoulli random variable that $\sigma^2 = \pi(1-\pi)$.  Thus we can rewrite the previous result as
$$ \frac{ \sqrt{n} \left( \hat{\pi} - \pi \right) }{ \sqrt{ \pi(1-\pi) } } \stackrel{L}{\rightarrow} N(0,1 ) $$
Furthermore, using the law of large numbers, it can be shown that $\hat{\pi} \stackrel{p}{\rightarrow} \pi$ (i.e. the estimate becomes arbitrarily close to the true value as the sample size $n$ gets larger and larger).  Thus, we may replace the true parameter values in the denominator by their sample values to obtain
$$ \frac{ \sqrt{n} \left( \hat{\pi} - \pi \right) }{ \sqrt{ \hat{\pi}(1-\hat{\pi}) } } \stackrel{L}{\rightarrow} N(0,1 ) $$
Finally to obtain a confidence interval we can do the following.  Suppose that we want a 95% confident interval.  Then the probability that a standard normal random variable has a value greater than -1.96 and less than 1.96 is 95%.  Therefore,
$$ \begin{align} 0.95 & \approx \mathbb{P}\left( -1.96 < \frac{ \sqrt{n} \left( \hat{\pi} - \pi \right) }{ \sqrt{ \hat{\pi}(1-\hat{\pi}) } } < 1.96 \right) \\[1ex]
& = \mathbb{P} \left( \hat{\pi} - 1.96\, \sqrt{\frac{ \hat{\pi}(1-\hat{\pi}) }{ n }} ~<~ \pi ~<~ \hat{\pi} + 1.96\, \sqrt{\frac{ \hat{\pi}(1-\hat{\pi}) }{ n }} \right) \end{align} $$
So in other words a 95% confidence interval is given by
$$ \hat{\pi} \pm 1.96\, \sqrt{\frac{ \hat{\pi}(1-\hat{\pi}) }{ n }} $$
Note however that this interval may include values less than zero or greater than one.  In that case, just truncate the interval to a lower bound of 0 or an upper bound of 1.
One final point to make.  There are several other methods of calculating a confidence interval for Bernoulli data which are thought to provide better coverage characteristics.  Some of these methods include a confidence interval based on the Rao score test, the Clopper-Pearson interval, and the mid-P-value interval.  But the interval described above is the default method and should do fine for a decent sample size.
