# Every second customer converts better. Can this be by chance?

At a bank customers are applying for a loan and then the customer gets a loan-offer which they can choose to accept or not. This bank has been running for 11 weeks and by coincidence a developer saw that odd applications-ids in average generate higher accepted amounts than even application-ids, every week. Application-id is just an incrementing number in the database, so has nothing to do with real world. There is no reason for why this would happen, so the question is, how likely is this to happen by chance? Someone said this can happen because splitting on odd/even application-id is not purely random. But isn't it random enough?

In the image you see the data:

• Column 1: week
• Column 2: group (even/odd application-id)
• Column 3: number of applications
• Column 4: number of accepted offers
• Column 5: sum of accepted amount
• Column 6: a flag if the accepted amount is bigger for odd application-ids (which it is for all 11 weeks)
• Column 7: A calculation by how much the accepted amount is bigger for odd application-ids.

With just the data you've posted, you can analyze it as a binomial distribution with each week being at trial. That would give a p-value of $$2^{-11}$$, or about $$0.05\%$$. You could also do a paired test, which would give you a binomial distribution with each pair of ids being a trial. Then there's a t-test on the total accepted amounts over odd ids and evens, but there's the issue of whether the underlying distribution is sufficiently normal.

Choosing what statistical test to do is problematic when you've already seen the data, as you'll be tailoring the test to match the pattern that you've already seen. However, there are different standards for when you're making a conclusion, versus when you're looking at whether something warrants further study. A p-value of $$0.02\%$$, even if it may be the result of p-hacking, justifies further investigation.

The data you've presented is very summary. You should get a more detailed look at the data. For instance, create a scatter plot with date of loan on one axis, size of loan (or perhaps log of size, if they are distributed over many orders of magnitude) on another, and different colors for odd or even ids.

There are a number of statistical test you could use to answer a question like this, but the simplest is the 2×2 chi-square contingency test (example here).

Just add up all the accepted applications with odd and even IDs across all the dates so that you have two numbers: $$N_{odd}, N_{even}$$. Call your random variable $$X \in \{0,1\}$$ where 0 means even and 1 means odd. The ex-ante probability of $$X=0$$ (given that odd/even is random) is $$p=0.5$$. You've got $$N=N_{odd}+N_{even}$$ trials and you want to know the probability that $$N_{even}$$ of them are even given $$p=0.5$$. So, the distribution you want is the Binomial distribution, and thus $$X\sim Binomial(N,p)$$.

If $$P(X=1\mid N,p) where $$x$$ is something suitably small, then you have a problem. If not then you could say it was chance.

• This is extremely confusing. "add up" implies sum, but $N_{odd}$ appears to be count of odd. Whether an id is even or odd is not a random variable. – Acccumulation Nov 27 '20 at 22:42
• @Accumulation sure youre right it was a bit confusing. I've added it to clarify that it refers to accepted applications. – emiru Nov 28 '20 at 9:33

At face value you have 11 samples in every case with B bigger than A. There are 2048 choices that could have happened (2^11) - viewing the choices as 11 binary bits makes it easy to visualise (assume bit is 0 for A>B and 1 for A<B). There is only one way B could always be bigger - the bit sequence 11111111111. But you had no a priori reason to think B would be bigger, so the equally extreme sequence 00000000000 is also equally valid. That would lead to a 1 in 1024 chance of the observed effect, highly significant (P~=0.001)

BUT as people have noted, you've only asked because you've seen an extreme pattern. So maybe 1023 other banks never asked.

AND look 2/3 of the way down your table. "B is bigger" seems to be untrue for 533 vs 527 (it is untrue for the ratios as well). So we've back to a lower probability, as any of the 11 could be 'reversed' and be just as extreme as your data. So that's another 22 cases that are just as extreme. So that's 24 of 2048, which is no longer significant at p=0.01.

Bottom line however - you're a new bank, it is quite an extreme pattern. I'd believe it was a true effect until I saw contradictory data. I don't know what programming language you're using but I'd suspect a reuse of a variable is causing the problem (or less likely a memory leak)

There is more information available in the actual ratios than in just saying whether A>B, but this is already enough of a smoking gun to look at the code - anything could happen statistically, it's not proof, the proof will be finding what's wrong with the code.

• This was a very useful answer, thanks! Regarding the 533 vs 527 you look at the wrong column, it’s for SUM_ACCEPTED_AMOUNT b is bigger. So I guess we’re back at 1/1024 then? – Lars Nov 29 '20 at 20:59