“The variation achieved a 20% increase in conversions with 90% significance.” Unfortunately, this isn’t equivalent to, “there is a 90% chance of a 20% increase.” So what does it actually mean?
20% is the increase we observed in our sample, and, if we had to guess, it’s the increase we’d expect to see if we continued the test indefinitely. It’s the most likely outcome. But there is not a 90% chance of a 20% increase, or even a 90% chance of an increase of at least 20% or approximately 20%.
90% is the chance that there would be a difference at all. In other words, if we had ten tests producing this result, and we decided to continue all ten tests indefinitely, we’d expect one to end up with the original and variation converging to identical conversion rates. Of the remaining nine tests, some might end up showing an increase of far less than 20%. Others might end up showing an even bigger increase.
I somehow feel that the 2nd paragraph is a poor explanation or maybe even wrong. The way I look at it, in simple words, it means there is only a 10% probability that the 20% variation was due to chance.