Doubts with P-value and Hypothesis in Permutation Test problem

Question

I have to solve this problem for a stats course:

An e-commerce company is testing a new design for a web page. The objective is to achieve at least a 2% increase in the conversion rate.

An experiment has been designed with two groups:

• Control Group A

• Treatment group B

The table with data has 30 rows and looks like this:

In order to solve this we are requested to define the hypotheses and test them with a permutation test, which I did.

I defined the hypotheses as follows:

• $H_0: \mu_{B} - \mu_{A} \geq 2\%$
• $H_1: \mu_{B} - \mu_{A} < 2\% $

And this were my results:

My P-value is almost 0, which rejects the null hypothesis.

I'm not sure if I'm formulating my hypotheses correctly as I actually want to confirm my $H_0$ instead of rejecting it and I have doubts if my P-value is being calculated correctly.

Help pls!

Answer 1

I would get the mean delta for B-A, subtract off 0.02, and called this $\mu_{obs}$. Then during $B=1000$ iterations, get the mean delta of B-A with column A shuffled (permuted), and call this $\mu^{(b)}$. The p-value will be equal to the number of times $\mu^{(b)}$ is greater than $\mu_{obs}$, divided by $B$. You know that when you shuffle column A and then take the difference between B and A, that average will probably be zero. You also know that B-A for the observed data (unshuffled), $\mu_{obs}$, may be 0.02 or more. Thus you have to subtract off the 0.02, and this way you will be testing that the two "zero means" means are significantly different.

See a recent post on permutation testing that I provided here.