3
$\begingroup$

I am creating a testing dashboard for displaying the results of our website testing. We run A/B tests and Multivariate tests. My testing code is sound (the gathering of the raw conversion data), but now I am working on finishing out our results reporting admin dashboard. (before we used a very old version of JMP to analyze our results but that was getting tedious so we/I decided to code up an admin dashboard with R)

Here is an example set of our data. (for an A/B test that we ran)

>a=c(1.3571428571429,1.3941176470588,1.275,1.2408759124088,1.5842696629213,
     1.5537634408602,1.4590909090909,1.4018691588785,1.495145631068,1.1794871794872,
     1.5114503816794,1.2569832402235,1.5336538461538)
>b=c(1.4891304347826,1.4656084656085,1.44,1.5190839694656,1.6084656084656,
     1.4166666666667,1.5665024630542,1.2533333333333,1.4081632653061,1.6173913043478,
     1.468253968254,1.5392670157068,1.5081081081081)

And with that data I am running the following R code to calculate the p-value / statistical significance.

>t.test(a, b, paired=TRUE)

Now after a week or two we would like to know how long we would need to continue to run the test before we would get to a given level of statistical significance, given the current data that we have collected.

I tried n.ttest but I have since been told that this is more meant for power analysis.

So my question is what R function should I use to get this information? Also on a related note I would be interested to know if there is any other calculations I should be running on our test data that we need that I could also add to our testing dashboard.

$\endgroup$
2
+50
$\begingroup$

What you are describing is essentially power analysis so power.t.test() would actually be one way to get at what you want. You can determine n in this way, given delta the effect size, or in your case the difference in the groups, and a given significance level.

"Given the data collected to date", means delta, n and sd, and this would look like:

delta <- mean(a-b) n.samp <- length(a) sd.tt <- sd(a-b)/sqrt(n.samp) power.t.test(delta = delta, power = 0.99, sd = sd.tt)

giving an n of 15, with power of 0.99 (1-type II error rate). This means that if you had the same delta and sd with n = 15, your t-test would give you a p-value = 0.05 or less.

If you wanted a significance level of say 0.001, you would need n = 26, as given by:

power.t.test(delta = delta, power = 0.99, sig.level = 0.001, sd = sd.tt)

HOWEVER, note that this kind of exercise should ONLY ever be conducted a priori, before you begin your experiment, using an effect size that makes sense in terms of the response you are measuring (you might choose a number of different possible effect sizes). You may do this exercise again once the testing is completed and you have found you cant reject your null hypothesis and to gain understanding of whether running the experiment for a longer period may have been beneficial (for next time).

You should not be running your experiment until your reach a level of significance you are happy with!

See the R help page for more details on the power.t.test() function http://stat.ethz.ch/R-manual/R-patched/library/stats/html/power.t.test.html

And google for more details on power analysis

$\endgroup$
  • $\begingroup$ I understand, yeah I guess I was aware that was the way you are suppose to do it. We just really have a hard time in knowing what the amount of change would be. Like if we are doing a pricing test and raise prices by 25%, how are you suppose to know what your effect size is/will be? $\endgroup$ – byoungb Jun 16 '14 at 14:11
  • $\begingroup$ It can be a useful exercise to look at this for a number of different effect sizes. Ultimately, the effect size needs to be about an understanding of what it is you are measuring. What are you measuring the effect of the 25% price increase on? What would you expect the effect to be and how pronounced? $\endgroup$ – carnust Jun 16 '14 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.