Am I taking crazy pills here or this A/B testing tool's documentation dead wrong? In the documentation here  for a popular A/B testing they say,

For example, if you run an A/B testing experiment with a significance
  level of 95%, this means that if you determine a winner, you can be
  95% confident that the observed results are real and not an error
  caused by randomness. It also means that there is a 5% chance that you
  could be wrong.

I believe this to be dead wrong.
Statistical significance at 95% means that $\text{p-value} < 0.05$. The p-value is defined as 
$$\text{p-value} \equiv P( \text{reject the null} \;|\; \text{the null is true})$$
The complement of this is $$1 - \text{p-value} = P( \text{fail to reject the null} \;|\; \text{the null is true})$$
So 95% significance tells you $P( \text{fail to reject the null} \;|\; \text{the null is true}) > 0.95$.
Assuming "determining a winner" means rejecting the null, the part of the documentation that says "if you determine a winner, you can be 95% confident that the observed results are real and not an error caused by randomness" to me means $0.95 > P(\text{null is false} | \text{reject the null})$ which is flatly not equivalent. 
Am I taking crazy pills here? Is there some reasonable way to convert whatever the heck they're saying with "95% confident that the observed results are real and not an error caused by randomness" into $P( \text{fail to reject the null} \;|\; \text{the null is true})$?
 A: You are right that the documentation is wrong.
Note that p values are defined somewhat differently from what you write. They do not measure the probability of a decision, such as the decision to reject the null or to fail to reject the null. They measure the probability of test statistics. Whether or not to reject a null hypothesis is a subsequent decision based on the p value and the alpha threshold.
Instead of test statistics, one often uses the shorthand "data":
$$\text{p-value} \equiv P( \text{data} \;|\; \text{the null is true}).$$
As you write, there is simply no way whatsoever to get from this to what the documentation is writing about, which is
$$\text{p-value}\; (\not\equiv)\; P( \text{the null is true} \;|\; \text{data}),$$
unless you are prepared to go the Bayesian route, posit priors etc. Not even "any charitable way (even if it's a stretch)". Nothing.
This is a very common misunderstanding. The American Statistical Association (ASA) recently published a statement on p values, and the statement we are discussing here is treated in statement 2 in the ASA document (see p. 131 in the The American Statistician document):

P-values do not measure the probability that the studied hypothesis is true

