I have to solve this problem for a stats course:

An e-commerce company is testing a new design for a web page. The objective is to achieve at least a 2% increase in the conversion rate.

An experiment has been designed with two groups:

  • Control Group A

  • Treatment group B

The table with data has 30 rows and looks like this:


In order to solve this we are requested to define the hypotheses and test them with a permutation test, which I did.

I defined the hypotheses as follows:

  • $H_0: \mu_{B} - \mu_{A} \geq 2\%$
  • $H_1: \mu_{B} - \mu_{A} < 2\% $

And this were my results:

enter image description here

My P-value is almost 0, which rejects the null hypothesis.

I'm not sure if I'm formulating my hypotheses correctly as I actually want to confirm my $H_0$ instead of rejecting it and I have doubts if my P-value is being calculated correctly.

Help pls!


1 Answer 1


I would get the mean delta for B-A, subtract off 0.02, and called this $\mu_{obs}$. Then during $B=1000$ iterations, get the mean delta of B-A with column A shuffled (permuted), and call this $\mu^{(b)}$. The p-value will be equal to the number of times $\mu^{(b)}$ is greater than $\mu_{obs}$, divided by $B$. You know that when you shuffle column A and then take the difference between B and A, that average will probably be zero. You also know that B-A for the observed data (unshuffled), $\mu_{obs}$, may be 0.02 or more. Thus you have to subtract off the 0.02, and this way you will be testing that the two "zero means" means are significantly different.

See a recent post on permutation testing that I provided here.

  • 2
    $\begingroup$ The real problem is that the OP is mixing up the null and alternate hypotheses. $\endgroup$
    – whuber
    Apr 14, 2021 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.