# Confidence in improvements when accuracy is not consistent

I have a real world problem. We run a business where we measure our success in number of payments monthly. We use Google Analytics which missing around 5-7% of our payments. So for example, one month we got 947 payments, but Google Analytics tracked 911.

Now, trying to grow the business, we A/B test parts of our web site. The problem is, that those tests can show anything from 1-N% improvement based on number of payments.

The question is, what's the improvement percentage we need to get in order to be sure it improved and that it's not just Google Analytics tracking a bit better this month?

I would also be satisfied if anyone point me in the right direction or tell me what is this related to (which topic).

• Do you know both the true number of sales and the google number for all months? Commented Sep 19, 2016 at 23:27
• Yes we do. In our database we have the real numbers. On GA we have tracked numbers with some error. Then the experiments we are running (A/B test) we only get numbers from GA. That's why the question. How do we go about trusting the experiments (A/B tests) Commented Sep 20, 2016 at 11:34
• The question I would ask first is: what is the reason that GA is not tracking certain payments?
– Jim
Commented Sep 20, 2016 at 15:06
• The problem is not about GA, because GA can't be 100% accurate anyway. Users can have some addons installed that block GA or plugins messing with it and so on... This we understand quite well. But even if it's 1-3% off and we get 2% improvement on our experiments how do we go about explaining that. Commented Sep 20, 2016 at 15:09
• @MatjazMuhic my point was more: is this process independent, or not? In the latter case you're sample would be biased.
– Jim
Commented Sep 20, 2016 at 15:16

This is not really a multivariate problem.

Basically it's a hypothesis testing problem. The basic question is "has the value improved?", or in statistical terms "has the expectation value changed".

In general when counting something, the Poisson distribution would be the right fit. Which means, that you have a uncertainty of the expectation value $\mu$ of the size $\sqrt\mu$. Here you have a different kind of uncertainty added on top of that -- the inaccuracy introduced by GA. So, if the number of participants in the A/B testing is high enough, you will still have a Poisson fluctuation plus the GA inaccuracy.

Do you have a way of knowing how large the uncertainty from GA is?

If yes, take that number an add in quadrature to the Poisson uncertainty (the variances add up); the two sources of uncertainty are independent. Then do the hypothesis testing.

Addition: if the total number of lost elements is known, but not their association -- e.g. the mismatch between the accounted number of transactions from the independent database, but not how much in type A or B -- then one can estimate the resulting uncertainty. One can make a guess to estimate this uncertainty as being proportional to the relative rate of A and B that GA delivers. The most simple assumption is that the number that GA deliver are consistent and unbiased -- so in short a fairly good estimator of the true values. On average this will be the true answer. One should note that the extreme cases (all elements are lost in only one case, and only in the lower count) are not represented by this. Very conservatively one could take these specific cases and also calculate the confidence levels for this and say that the "true" answer lies somewhere in between. If the numbers are large enough, the difference will be small.

If none of the above, think of a good number and do the same. If your assumed error is too large, you will see an improvement only when it's large enough. If it's too small, you might trick yourself into believing it's become better.

As a side note: the uncertainties are binomial, but if the number of counts is high enough the Poisson distribution is a good approximation. At some stage, at least for central regions Gaussian distribution will be good enough as well.

• Well, the thing that is testing variations is GA so the numbers about the original version are also not 100%. Does that change anything? Commented Sep 22, 2016 at 17:35
• I don't understand what you mean with "numbers about the original version". If I understand correctly, then you want to do hypothesis testing with data that has a certain uncertainty. The difference to a textbook case seems to me that you have two sources of uncertainty: one from having limited statistics, which is approximately Poissonian and another one from the fact that you don't know the actual number. You could try to model the uncertainties better, taking even possible correlations into account. But you still won't know the real number. Commented Sep 22, 2016 at 18:43
• That's true, but we do know the accurate number of all payments from a different source (our database). Does that help at all? So the way I was looking at it is like this. We have 110 payments recorded in our database (100% accurate). GA lost 10 payments. Now, original variant has 30 payments, A variant has 32 payments and B variant has 38 payments. We don't know where we lost those 10 payments. Now GA will tell us variant B is 26% better. We know GA lost around 10% but we are not sure if on all variants or one is more distorted. Commented Sep 23, 2016 at 7:19
• Your guess for the uncertainty of the GA is much better, when you know the number/fraction of lost entries. But the uncertainty of where is not reducible. As said above, you could put some effort into modelling the GA uncertainty. Probably it's fair to say, that the GA loss is unbiased -- at least according to big Gs motto of not being evil. So your best guess is that the "lost numbers" are distributed like the ratio that is actually counted - e.g. 10 are lost, A has 40, B has 60. So the uncertainty becomes $\sigma_A = \sqrt{40 + 4}$ and $\sigma_B = \sqrt{60 + 6}$. Commented Sep 23, 2016 at 9:38
• A ok I see. That makes sense! Commented Sep 23, 2016 at 12:09

I understand your challenge is that you can track the A/B path only through the google analytics (GA) What you can do then is to measure the standard deviation of the overall GA error and add it to the calculation of the confidence interval.

EDIT: I see that @cherub has added an answer to the same effect earlier.

• We do know the accurate number of all payments from a different source (our database). Does that help at all? So the way I was looking at it is like this. We have 110 payments recorded in our database (100% accurate). GA lost 10 payments. Now, original variant has 30 payments, A variant has 32 payments and B variant has 38 payments. We don't know where we lost those 10 payments. Now GA will tell us variant B is 26% better. We know GA lost around 10% but we are not sure if on all variants or one is more distorted. Commented Sep 23, 2016 at 7:20
• So you actually have three variants: original, A and B, correct? Commented Sep 23, 2016 at 13:29
• Yes, but we could have more than that. Depending on the experiment/test. Commented Sep 23, 2016 at 13:38