So I have setup testing code on a website, our conversion data is blocked per day and fed into R.
This is the R code that I am using to calculate the "confidence"
t.test(a, b, paired=TRUE)
And this is the R code that I am using to calculate "sample size"
n.ttest(power = 0.8, alpha = 0.05, mean.diff = (mean(a) - mean(b)), sd1 = sd(a), sd2 = sd(b), k = 1, design = "paired", fraction = "balanced", variance = "equal")
Everything was looking really good, until one of our tests had confidence of 95% after 15 days even though the sample size required to get to 80% on the same data said it would take 30 days. This confused us because we thought they were related more strongly then that.
Our current guess is that this is caused by our weekends having about half the conversions of normal weekdays.
So my real question is, am I doing this correct and if so does the slow weekends make sense as a reason for the elevated sample size or is there a better way altogether?
Update So I took @jsk's advice because he is right, n.ttest when paired/equal only uses sd1. But my numbers are still off. Here is the actual R code and output.
>a=c(1.3571428571429,1.3941176470588,1.275,1.2408759124088,1.5842696629213, 1.5537634408602,1.4590909090909,1.4018691588785,1.495145631068,1.1794871794872, 1.5114503816794,1.2569832402235,1.5336538461538) >b=c(1.4891304347826,1.4656084656085,1.44,1.5190839694656,1.6084656084656, 1.4166666666667,1.5665024630542,1.2533333333333,1.4081632653061,1.6173913043478, 1.468253968254,1.5392670157068,1.5081081081081)
> t.test(a, b, paired=TRUE) Paired t-test data: a and b t = -1.657, df = 12, p-value = 0.1234 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -0.18824254 0.02560796 sample estimates: mean of the differences -0.08131729
> n.ttest(power = 0.8, alpha = 0.05, mean.diff = (mean(a) - mean(b)), sd1 = sd(a - b), k = 1, design = "paired", fraction = "balanced", variance = "equal") $`Total sample size`  40
So as you can see I get a p-value of 0.1234 with a sample size of 13 data points, but the sample size calculations says that it would require 40 data points so what exactly is "power" and "alpha" and how do they relate to the p-value. For instance what would I have to set it to the know the sample size to get to a p-value of 95%??