Confidence in improvements when accuracy is not consistent I have a real world problem. We run a business where we measure our success in number of payments monthly. We use Google Analytics which missing around 5-7% of our payments. So for example, one month we got 947 payments, but Google Analytics tracked 911.
Now, trying to grow the business, we A/B test parts of our web site. The problem is, that those tests can show anything from 1-N% improvement based on number of payments.
The question is, what's the improvement percentage we need to get in order to be sure it improved and that it's not just Google Analytics tracking a bit better this month?
I would also be satisfied if anyone point me in the right direction or tell me what is this related to (which topic).
 A: This is not really a multivariate problem.
Basically it's a hypothesis testing problem. The basic question is "has the value improved?", or in statistical terms "has the expectation value changed".
In general when counting something, the Poisson distribution would be the right fit. Which means, that you have a uncertainty of the expectation value $\mu$ of the size $\sqrt\mu$. Here you have a different kind of uncertainty added on top of that -- the inaccuracy introduced by GA. So, if the number of participants in the A/B testing is high enough, you will still have a Poisson fluctuation plus the GA inaccuracy.
Do you have a way of knowing how large the uncertainty from GA is? 
If yes, take that number an add in quadrature to the Poisson uncertainty (the variances add up); the two sources of uncertainty are independent. Then do the hypothesis testing. 
Addition: if the total number of lost elements is known, but not their association -- e.g. the mismatch between the accounted number of transactions from the independent database, but not how much in type A or B -- then one can estimate the resulting uncertainty. One can make a guess to estimate this uncertainty as being proportional to the relative rate of A and B that GA delivers. The most simple assumption is that the number that GA deliver are consistent and unbiased -- so in short a fairly good estimator of the true values. On average this will be the true answer.
One should note that the extreme cases (all elements are lost in only one case, and only in the lower count) are not represented by this. Very conservatively one could take these specific cases and also calculate the confidence levels for this and say that the "true" answer lies somewhere in between. If the numbers are large enough, the difference will be small.
If none of the above, think of a good number and do the same. If your assumed error is too large, you will see an improvement only when it's large enough. If it's too small, you might trick yourself into believing it's become better.
As a side note: the uncertainties are binomial, but if the number of counts is high enough the Poisson distribution is a good approximation. At some stage, at least for central regions Gaussian distribution will be good enough as well.
A: I understand your challenge is that you can track the A/B path only through the google analytics (GA)
What you can do then is to measure the standard deviation of the overall GA error and add it to the calculation of the confidence interval. 
EDIT: I see that @cherub has added an answer to the same effect earlier.
