Is my interpretation of the p-value and confidence interval correct? Background: 
let's say I have a website with a big buy button on it and I preformed an A/B testing with the following result: 
Version A (control): conversion rate 5%
Version B (variation): conversion rate 8%
Significant level:   5%
Confidence interval:  +/- 1%, (8%-5%) = 3% thus the bound is (2%, 4%)
p-value: 0.02
Null hypothesis: the conversion rate of the control group is the same as the conversion rate of the variation.
Alternative hypothesis：the conversion rate of the control group is different from the conversion rate of the variation.
I made up the numbers above but here is how I would calculate these values:
$$ z = \frac{P_{A} - P_{B}} {\sqrt{\frac{P_A(1-P_A)}{n_A} + \frac{P_B(1-P_B)}{n_B}}} $$
$P_A$: the conversion rate for the control
$P_B$: the conversion rate for the variation
$n_A$: the number of visitors who saw the control
$n_B$: the number of visitors who saw the variation  
The formula above is based on the independent two-sample t-test:
$$ t= \frac{\bar{X_A} - \bar{X_B}}{\sqrt{\frac{{S_A}^2}{n_A} + {\frac{{S_B}^2}{n_B}}}}$$
Because click conversion rate is essentially a Bernoulli trial, so it follows the Bernoulli distribution.
With Bernoulli distribution, the variance is calculated as $p(1-p)$, thus replacing ${S_A}^2$ with $p(1-p)$ gives the formula for calculating $z$ above.

My interpretation of the p-value: 
Assuming there is no difference between the conversion rate of the control and variation, there is only a 2% chance (1 - p-value) that we observe a difference like we have seen. Thus, there are two explanations:


*

*We've only experimented once and we observed something that only happens less than 5% of the time. Theoretically speaking, it is possible to get such extreme data on the first try. If I reject the null hypothesis incorrectly, I am running an extreme low risk of making the type I error.

*We've only experimented once and we observed something that only happens less than 5% of the time, so this must not be luck and we should conclude that the difference is real. Thus, the variation group is indeed different from the control group.

My interpretation of the confidence interval：
If we were to repeat the same experiment 100 times (with 100 different group of people testing both versions), then we should expect to see the true difference between the two groups appear in 95 of those experiments.
We think we are the 95%, and if that is true, then the true difference lay somewhere between 2% and 4%.
But there is also a 5% chance that the range (2%, 4%) completely misses the true difference.
We could also be the 5%, such that the range (2%, 4%) does not contain the true difference.

Is my interpretation of the p-value and confidence interval correct?
 A: Your last claim is wrong. There is not a 5% chance that your range 2% to 4% misses the mark. It either misses or it does not, 0% or 100%; you just don't know.
The p-value also assumes you met the assumptions of the statistical test you conducted if you are being pedantic.
For claim number 2, you shouldn't ever conclude definitively from one study. But you have evidence to suggest that the difference is there.
Additionally, this decision to claim there is a difference only happens less than 5% of the time (given an $\alpha$ of .05) only if the effect is literally zero.

The p-value is not the probability of making a type-I error. Before a study is conducted, the chosen $\alpha$-level is. After the study is conducted, you either have made or you have not made a type-I error.
I might rewrite 1 thus:
We've only experimented once and we observed something that only happens less than 5% of the time if the null is true. I may be in the 5% of people who will make an error of claiming a difference from the null when there is none. Thus, if I reject the null, I may have made a type-I error.
I might rewrite 2 thus:
We've only experimented once and we observed something that only happens less than 5% of the time if the null is true. Hence, this is evidence to suggest that the null is unlikely to be true. And we may conclude that there is a difference between the control group and variation group that is not due to sampling error.
