Binomial random variables versus chi-square for A/B testing? I've been reading on how to calculate confidence levels from A/B tests, as I'm running some ads on Facebook and want to better understand the results. The clearest explanations I've found seem to offer two alternatives: Chi-Square and Binomial Random Variables.  I'm hoping someone can explain which is more appropriate.
I ran two ads on Facebook:
Ad    Impressions    Clicks    CTR       
1     58,028         3         0.005%    
2     189,724        24        0.013%    

According to this Chi-Square calculator:
http://www.usereffect.com/split-test-calculator
I have not yet achieved statistical significance, and will need over 40k more views to achieve a 90% confidence level.
This site explains Binomial Random Variables:
http://visualwebsiteoptimizer.com/split-testing-blog/what-you-really-need-to-know-about-mathematics-of-ab-split-testing/
According to that math, both ads have a very, very small standard error, and so I can trust that the Click-through Rate is pretty accurate.
Any help would be appreciated.
 A: The terminology of this is unfamiliar to me.  But I gather you are looking for evidence to dismiss the null hypothesis that Ad 1 has the same proportion of Clicks to Impressions as Ad 2 (if you're interested in the total number of impressions or clicks, none of the below is appropriate).
When you have these extreme proportions, something that works directly with the distribution of binomial variables is definitely preferable to a Chi square ($\chi^2$) approximation - but don't be fooled into thinking there are no approximations with the binomial tests.  The common label "exact" applied to binomial tests is quite misleading as you still need to estimate the key parameters.
It is true that the standard errors of binomial variables with very small proportions are low, but a problem is that you don't know the real proportion - you only have an estimate of it.  Hence you only have an estimate of the variance that comes with the proportion and hence the standard error of your estimate of it (and if you're thinking in terms of standard errors, this probably means you are about to stop working directly with the binomial distribution and start using Normal approximations, anyway).  
While you have a huge number of trials you have only a small number of successes and a few either way make quite a difference (particularly for Ad 1).
But most importantly, maybe you have two "pretty accurate" estimates of your click through rates, but they are still reasonably similar to eachother.  You do need more data to be confident they are different.  There are various ways of testing this or of producing confidence intervals for your two click through rates, but I'm pretty sure they'll all return similar results.
Sorry, I realise the above isn't that clear... I'm happy for someone to edit it to make it better!
I ran the following in R, using Frank Harrell's Hmisc library.
library(Hmisc)
ads <- data.frame(binconf(c(3,24), c(58028,189724), method="wilson") *100 )
ads$ad <- c("One", "Two")
ads
qplot(ymin=Lower, ymax=Upper, x=ad, data=ads, 
    geom="errorbar") + labs(y="Click through percentage")  +
    geom_point(aes(y=PointEst))

This gives you 
     PointEst       Lower      Upper  ad
1 0.005169918 0.001758254 0.01520047 One
2 0.012649955 0.008501270 0.01882285 Two

and the plot (much easier to see the overlapping confidence intervals in a plot - including how the confidence intervals are asymmetric around your point estimate, which is something that happens when you have proportions close to 0 or 1).

