How to determine reliability of a change in an arbitary metric I have been using a/b testing in order to optimise changes I make to my websites.  Previously, I have set a single conversion goal and have ended up with a conversion rate for each group.  I then used R (with the power.prop.test function) to calculate the power and significance of the change in conversion rate to decide when it is safe to stop the test and be confident that the values have meaning.
I am now able to get information on a wide range of metrics about the site for each group.  However I am not sure how to do the same test for an arbitrary metric - e.g., revenue.  Could someone point me in the right direction?  I am happy to use R so just pointing me at the right set of functions would be great, but I do like to try and understand what I am calculating so any explanations or links to useful site would be even better.
 A: The simplest method I can recommend in this case is permutation testing. You basically have two groups of values (A and B) and you want to know if B is meaningfully larger than A. Let's say you can measure this as the distance between the mean values of A and B. (This is usually clinically meaningful in the context you mention.)

*

*The null hypothesis is that A and B are "equivalent" which means that we can combine A and B values into a single data set, from which we draw (without replacement) two new groups of values $A_p$ and $B_p$.


*Compute differences of the means of $A_p$ and $B_p$.


*Repeat the above two steps a large number of times. From this step, you get an entire distribution of differences of means based on the null hypothesis.


*Check how often the values in the distribution from the previous step are equally or more extreme than the one you observed in real life.
The result of 4 is how confident you can be in your result.
I gave this example in terms of "differences of means" but it works for any function improvement(A,B) that measures how much B improves over A. You simulate a bunch of A and B sets from the data you have and a null hypothesis that A and B are equivalent, compute the improvement for each simulated pair, and then see how often that is more extreme than the one you've seen.

Another option is the bootstrap:

*

*From A, draw with replacement a new bootstrap replication $A_b$. Do the same with B, getting a bootstrap replication $B_b$.


*Compute the improvement on the replicated pair of datasets.


*Repeat the above a large number of times. You will get out of it a distribution of improvements that are all plausible given the observed distributions for A and B.


*For e.g. a confidence level of 95 %, check if the largest 95 % of values the distribution of improvements from the previous step include 0. If it does, you have not observed a significant effect.
This is in some ways more indirect (you are using bootstrap replications to approximate the sampling distribution of the underlying problem, instead of showing what happens under the null distribution) but can also be easier to implement properly.
This code was written on my phone without testing it, so it probably doesn't work, but should get you started!
# assuming vectors A and B exist already, as well as a function
# improvement(A,B) that returns a number indicating
# the size of the improvement.

distr <- rep(NA, 5000)
for (i in 1:length(distr)) {
    distr[i] <- improvement(
        sample(A, length(A), replace=T),
        sample(B, length(B), replace=T)
    )
}

quantile(distr, probs=0.05)

This will give you the smallest value of the largest 95 % in the distribution. If that is greater than zero, you have a significant effect at the 0.05 level!
