Website (A/B or Multivariate) testing, Statistical significance not lining up with sample size

So I have setup testing code on a website, our conversion data is blocked per day and fed into R.

This is the R code that I am using to calculate the "confidence"

t.test(a, b, paired=TRUE)


And this is the R code that I am using to calculate "sample size"

n.ttest(power = 0.8, alpha = 0.05, mean.diff = (mean(a) - mean(b)), sd1 = sd(a),
sd2 = sd(b), k = 1, design = "paired", fraction = "balanced", variance = "equal")


Everything was looking really good, until one of our tests had confidence of 95% after 15 days even though the sample size required to get to 80% on the same data said it would take 30 days. This confused us because we thought they were related more strongly then that.

Our current guess is that this is caused by our weekends having about half the conversions of normal weekdays.

So my real question is, am I doing this correct and if so does the slow weekends make sense as a reason for the elevated sample size or is there a better way altogether?

Update So I took @jsk's advice because he is right, n.ttest when paired/equal only uses sd1. But my numbers are still off. Here is the actual R code and output.

Data

>a=c(1.3571428571429,1.3941176470588,1.275,1.2408759124088,1.5842696629213,
1.5537634408602,1.4590909090909,1.4018691588785,1.495145631068,1.1794871794872,
1.5114503816794,1.2569832402235,1.5336538461538)
>b=c(1.4891304347826,1.4656084656085,1.44,1.5190839694656,1.6084656084656,
1.4166666666667,1.5665024630542,1.2533333333333,1.4081632653061,1.6173913043478,
1.468253968254,1.5392670157068,1.5081081081081)


Confidence

> t.test(a, b, paired=TRUE)
Paired t-test
data:  a and b
t = -1.657, df = 12, p-value = 0.1234
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.18824254  0.02560796
sample estimates:
mean of the differences
-0.08131729


Sample Size

> n.ttest(power = 0.8, alpha = 0.05, mean.diff = (mean(a) - mean(b)), sd1 = sd(a - b),
k = 1, design = "paired", fraction = "balanced", variance = "equal")
$Total sample size [1] 40  So as you can see I get a p-value of 0.1234 with a sample size of 13 data points, but the sample size calculations says that it would require 40 data points so what exactly is "power" and "alpha" and how do they relate to the p-value. For instance what would I have to set it to the know the sample size to get to a p-value of 95%?? • Wow thanks, I wasn't sure where I should post this, I use stack overflow every day, but I do not know much about the rest of the sites. – byoungb Apr 29 '14 at 13:18 • I assume that when you say that your test had 95% confidence after 15 days you mean that the p-value was less than .05? – jsk May 1 '14 at 7:03 • Correct, we tried to simplify the lingo a little. – byoungb May 1 '14 at 13:14 2 Answers You have a misconception of power here. When you do a power calculation, you assume that something is true. In this case, you're assuming > mean(a-b) [1] -0.08131729 > sd(a-b) [1] 0.1769424  is the true state of the world. Now having assumed this, a power calculation asks how large of a sample is needed so that you have$p$probability of rejecting the hypothesis that the effect/difference is zero at a specified$\alpha$level. Your actual experiment might (and will) return a different point estimate than your assumed state of the world, but that's beside the point. The point of the power analysis is to say: "If these assumptions are true (which they probably aren't), we have a$p$chance at finding an effect far enough from zero that we can reject the null hypothesis if we have$n$samples, so let's spend some money!" In this case, you've set$p=0.8$and$\alpha=0.05$, and your power calculation returns 40. So if you run an experiment with 40 samples, you have an 80% chance of going "aha, rejected that the effect size is zero." This is confirmed with a simulation p1 = NULL p2 = NULL for (i in 1:10000) { # assume that this is really the state of the world x1 = rnorm(40, mean(a-b), sd(a-b)) x2 = rnorm(20, mean(a-b), sd(a-b)) # paired t-test equivalent to one-sample t-test of difference against zero t1 = t.test(x1) t2 = t.test(x2) p1 = append(p1,t1$p.value)
p2 = append(p2,t2$p.value) } > length(p1[p1<0.05])/length(p1) [1] 0.8139 > length(p2[p2<0.05])/length(p2) [1] 0.5016  Note that even though the simulation assumes that the effect$=-0.08$, we are only able to reject that the effect$\neq 0$slightly more than ~80% of the time, just as the power analysis showed. Running with fewer samples, the number of times you are able to reject the null decreases. In this case, if you had collected 20 samples instead of 40, you would have only managed to reject that the effect is non-zero 50% of the time. It's not clear whether your$a$and$b$are your final results, or if that's part of a preliminary study. Usually power analysis is done before an experiment is carried out, to basically make sure enough is collected so its not a waste of time. Some people do preform power analysis after the fact as a kind of "what went wrong" sort of deal. In this case, if this is your final data, note that your power, even assuming that the true effect$=-0.08\$, your power is only 33%
power.t.test(n=length(a),delta=mean(a-b),sd=sd(a-b),power=NULL, type="paired")

• Okay, I am really regretting not paying attention in my college statistics class. So I guess I am unclear what "power" actually is. What I am wanting is that I have a test running and I calculate the p-value on the data collected so far, but also would like to know how long or how many data points I will need to get a specific p-value, using that data, is this possible?? – byoungb May 7 '14 at 14:01
• @byoungb power is generally calculated before you run your experiment so that you can plan your study before you start it! Now that you have clarified your actual question, you should start a new question! – jsk May 9 '14 at 2:56
• stats.stackexchange.com/questions/97058/… – byoungb May 9 '14 at 14:02

The documentation for n.ttest is lacking, but it appears that if you desire to calculate the power for a paired t-test with the function n.ttest, then sd2 is ignored by the function and you should enter sd(a-b) for sd1. After all, there is only one sd of concern when performing a paired t-test, the standard deviation of the differences.

• I will look into it but this makes a lot of sense. (I will accept this as soon as check it out) Thanks! – byoungb May 1 '14 at 13:18
• Hmmm, that did change things but actually the data sample size went up not down. (I did look at n.ttest source and you are right it only uses sd1) I will update my question with more details. – byoungb May 6 '14 at 13:51