How does a p-value calculator work with only four inputs?

Question

I see lots of online p-value calculators for conversion rate, like this one: http://www.experimentationhub.com/p-value.html How can that work, without knowing the variance/SD?

I believe it's because conversion rate (i.e. the percentage of visitors to a web site who bought something) is just the average of a dummy (Boolean) variable (i.e. did the visitor buy something or not). And you can determine the variance of a dummy variable just from knowing its average, is that right?

So to compute the p-value of any "rate" (e.g. conversion rate, click-through rate, etc) you don't need access to the underlying samples, just the average and count. But for anything that's a number (e.g. order size in dollars) you'd need to know the SD too.

Answer 1

Yes, your understanding is essentially correct.

When simply comparing rates or proportions with only one independent variable (Group or whatever) then all the information you have about a subject is group and whether they did whatever it is you are measuring (convert, in your case).

Sometimes, though, you will have more information and, to incorporate that, you will need the individual data. E.g. you might want to incorporate demographic information into your analysis of conversion. Then you would probably want to do logistic regression.

Answer 2

It may be helpful to look at a chi-square test of association. It uses only counts, arranged in a table, to see if there an association between the variable indicated by the rows and the one indicated by the columns.

It looks like the page you cited might be using either an uncorrected chi-square test or uncorrected proportion test. (I don't think they tell you.)

You can run the following code in R, for example at R Fiddle or by downloading the software.

M = matrix(c(100, 130, 5000-100, 5000-130), ncol=2)

M

chisq.test(M, correct=F)

prop.test(M, correct=F)