I have the following data:
What can I do to check if there is a significant increase of the ratio of people paying for the product over the total visitings of the website?
I thought of a Chi-Squared test of Independence, but is it right? Can it be done with A/B testing? Are there any other options?
Confidence intervals. 95% confidence intervals for percent paid are:
$(.0205,.0247)$ for Control,
$(.0226,.0297)$ for New 1,
$(.0235,.0308)$ for New 2 (note overlap with New 1), and
$(.0241,.0292)$ for New Combined (narrower on account of larger combined sample size).
The formula for the Wald 95% confidence interval (used above) is $\hat p \pm 1.96\sqrt{\frac{\hat p(1-\hat p)}{n}},$ where $\hat p = X/n.$ (For samples this large, the Agresti-Coull or 'plus=4' correction makes no important difference.)
Tests of binomial proportions. If you do a test comparing Control vs. New Combined as independent proportions or Fisher's Exact test, you will find a significant difference.
From Minitab statistical software:
Test and CI for Two Proportions
Sample X N Sample p
1 425 18789 0.022620
2 411 15412 0.026668
Difference = p (1) - p (2)
Estimate for difference: -0.00404791
95% CI for difference: (-0.00736298, -0.000732844)
Test for difference = 0 (vs ≠ 0): Z = -2.39 P-Value = 0.017
Fisher’s exact test: P-Value = 0.017
I wouldn't necessarily combine the data for the two new websites as @BruceET suggests. There's some discussion under their response.
CONFIDENCE INTERVALS
I like @BruceET 's approach of looking at the confidence intervals for each proportion. It is easy and clear to present to an audience a plot of the proportion and confidence interval for each group.
Looking at these, I would encourage your audience to not get too hung up about a magic cutoff value of p = 0.05, or perfectly non-overlapping confidence intervals. The proportions and confidence intervals suggest the newer websites are similar and perhaps better than the old site.
LOGISTIC REGRESSION
The way I would probably approach this problem is with logistic regression. This produces clear results, that will also be easy to present, but the analysis may be less familiar for your audience.
The following example can be run in R.
Lets start by calculating the non-paying counts
Total = c(18789, 7842, 7570)
Payment = c( 425, 205, 206)
Nonpay = Total - Payment
Nonpay
### [1] 18364 7637 7364
Install some required packages
if(!require(car)){install.packages("car")}
if(!require(emmeans)){install.packages("emmeans")}
Input the data and run the logistic regression. The p value for the effect of Website is shown. It turns out it's just shy of meeting a 0.05 threshold. (And we still shouldn't get too hung up on the 0.05 threshold).
Data = read.table(header=T, text="
Website Pay Weight
Old Yes 425
Old No 18364
New1 Yes 205
New1 No 7637
New2 Yes 206
New2 No 7364
")
model= glm(Pay ~ Website, weights=Weight, data=Data,
family=binomial(link="logit"))
library(car)
Anova(model, test="Wald")
### Analysis of Deviance Table (Type II tests)
###
### Df Chisq Pr(>Chisq)
### Website 2 5.9893 0.05006
If we wanted to compare pairwise among the individual websites, we could use estimated marginal means. If we translate these marginal means from the analysis back to their original, "response", scale, we will get estimates for the proportions and their confidence intervals. Note that in this case, everything is quite close to those given by @BruceET .
library(emmeans)
marginal = emmeans(model, ~ Website, type="response")
marginal
### Website prob SE df asymp.LCL asymp.UCL
### New1 0.0261 0.00180 Inf 0.0228 0.0299
### New2 0.0272 0.00187 Inf 0.0238 0.0311
### Old 0.0226 0.00108 Inf 0.0206 0.0248
###
### Confidence level used: 0.95
### Intervals are back-transformed from the logit scale
Now, pairwise comparisons among individual websites with a Tukey adjustment for multiple comparisons.
pairs(marginal)
### contrast odds.ratio SE df z.ratio p.value
### New1 / New2 0.96 0.0960 Inf -0.413 0.9104
### New1 / Old 1.16 0.0999 Inf 1.722 0.1969
### New2 / Old 1.21 0.1040 Inf 2.204 0.0705
###
### P value adjustment: tukey method for comparing a family of 3 estimates
### Tests are performed on the log odds ratio scale
If we wanted to look at these comparisons without the adjustment.
pairs(marginal, adjust="none")
### contrast odds.ratio SE df z.ratio p.value
### New1 / New2 0.96 0.0960 Inf -0.413 0.6798
### New1 / Old 1.16 0.0999 Inf 1.722 0.0850
### New2 / Old 1.21 0.1040 Inf 2.204 0.0275
CHI-SQUARE TEST OF ASSOCIATION
Another approach is to arrange the counts in a contingency and use a chi-square test of association. Here the p value is just less than 0.05, so quite close to the previous analysis.
Input =("
Website Pay Nonpay
Old 425 18364
New1 205 7637
New2 206 7364
")
Matrix = as.matrix(read.table(textConnection(Input),
header=TRUE,
row.names=1))
Test = chisq.test(Matrix)
Test
### Pearson's Chi-squared test
###
### X-squared = 6.0033, df = 2, p-value = 0.0497
You might find some different different methods for post-hoc testing of chi-square test of association. Here, I'll suggest one approach: examining the standardized residuals. A standardized residual > 1.96 or < -1.96 corresponds to a p value of 0.05, and suggests that that cell is "interesting" relative to the expected values. Here, the cells in the Old row meet this criteria, suggesting that the cells in the Old row stand out "significantly" in the table.
Test$stdres
### Pay Nonpay
### Old -2.412033 2.412033
### New1 1.108878 -1.108878
### New2 1.767987 -1.767987
I'd go for a one-sided paired difference test to test whether the difference in ratios is statisticially different from zero. In your case, you would make use of the t-distribution instead of the normal one. However, they are asymptotically the same. Then you could test two differences in ratios ($r$): $r_{new_1} - r_{old} > 0$ and $r_{new_2} - r_{old} > 0$ for both new websites.
