2
$\begingroup$

“The variation achieved a 20% increase in conversions with 90% significance.” Unfortunately, this isn’t equivalent to, “there is a 90% chance of a 20% increase.” So what does it actually mean?

20% is the increase we observed in our sample, and, if we had to guess, it’s the increase we’d expect to see if we continued the test indefinitely. It’s the most likely outcome. But there is not a 90% chance of a 20% increase, or even a 90% chance of an increase of at least 20% or approximately 20%.

90% is the chance that there would be a difference at all. In other words, if we had ten tests producing this result, and we decided to continue all ten tests indefinitely, we’d expect one to end up with the original and variation converging to identical conversion rates. Of the remaining nine tests, some might end up showing an increase of far less than 20%. Others might end up showing an even bigger increase.

I somehow feel that the 2nd paragraph is a poor explanation or maybe even wrong. The way I look at it, in simple words, it means there is only a 10% probability that the 20% variation was due to chance.

Original Link here

$\endgroup$
2
$\begingroup$

Indeed, the second paragraph is wrong.

Probably a better way to rephrase the initial sentence would be: "The variation achieved a 20% increase in conversions. This increase was significant at the 0.1 level."

This means that if there was no real difference at all (increase = 0%), you would still expect 10% of the times you repeat your sample to get an increase at least as large as that 20% by chance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.