4
$\begingroup$

I have the following data:

enter image description here

What can I do to check if there is a significant increase of the ratio of people paying for the product over the total visitings of the website?

I thought of a Chi-Squared test of Independence, but is it right? Can it be done with A/B testing? Are there any other options?

$\endgroup$

3 Answers 3

1
$\begingroup$

Confidence intervals. 95% confidence intervals for percent paid are:

  • $(.0205,.0247)$ for Control,

  • $(.0226,.0297)$ for New 1,

  • $(.0235,.0308)$ for New 2 (note overlap with New 1), and

  • $(.0241,.0292)$ for New Combined (narrower on account of larger combined sample size).

The formula for the Wald 95% confidence interval (used above) is $\hat p \pm 1.96\sqrt{\frac{\hat p(1-\hat p)}{n}},$ where $\hat p = X/n.$ (For samples this large, the Agresti-Coull or 'plus=4' correction makes no important difference.)

Tests of binomial proportions. If you do a test comparing Control vs. New Combined as independent proportions or Fisher's Exact test, you will find a significant difference.

From Minitab statistical software:

Test and CI for Two Proportions 

Sample    X      N  Sample p
1       425  18789  0.022620
2       411  15412  0.026668

Difference = p (1) - p (2)
Estimate for difference:  -0.00404791
95% CI for difference:  (-0.00736298, -0.000732844)
Test for difference = 0 (vs ≠ 0):  Z = -2.39  P-Value = 0.017

Fisher’s exact test: P-Value = 0.017
$\endgroup$
2
  • $\begingroup$ I would assume that in this case it wouldn't make much sense to pool the the two new websites. For example, if the old site used a blue button, one new site used a red button, and one new site used a green button. But there may be some common feature that allows for pooling of the two new websites. That's up to the O.P... $\endgroup$ Nov 7, 2019 at 12:03
  • $\begingroup$ Lacking info, it's just as reasonable to guess new sites may be sight variations on a theme. Especially because of similar performance. // Obvious motivations for pooling: (a) No significance btw new sites. (b) Taken separately, neither new site is a convincing improvement. (c) Seems worthwhile to recognize that somehow new sites are better. // Agree it's up to OP for now. If this is a real situation, maybe wait for more data from new sites, which may provide resolution. // Do you have improved analysis to suggest? $\endgroup$
    – BruceET
    Nov 7, 2019 at 18:46
1
$\begingroup$

I wouldn't necessarily combine the data for the two new websites as @BruceET suggests. There's some discussion under their response.

CONFIDENCE INTERVALS

I like @BruceET 's approach of looking at the confidence intervals for each proportion. It is easy and clear to present to an audience a plot of the proportion and confidence interval for each group.

Looking at these, I would encourage your audience to not get too hung up about a magic cutoff value of p = 0.05, or perfectly non-overlapping confidence intervals. The proportions and confidence intervals suggest the newer websites are similar and perhaps better than the old site.

LOGISTIC REGRESSION

The way I would probably approach this problem is with logistic regression. This produces clear results, that will also be easy to present, but the analysis may be less familiar for your audience.

The following example can be run in R.

Lets start by calculating the non-paying counts

Total   = c(18789, 7842, 7570)
Payment = c(  425,  205,  206)
Nonpay  = Total - Payment
Nonpay

   ### [1] 18364  7637  7364

Install some required packages

if(!require(car)){install.packages("car")}
if(!require(emmeans)){install.packages("emmeans")}

Input the data and run the logistic regression. The p value for the effect of Website is shown. It turns out it's just shy of meeting a 0.05 threshold. (And we still shouldn't get too hung up on the 0.05 threshold).

Data = read.table(header=T, text="
Website  Pay  Weight
Old      Yes     425
Old      No    18364
New1     Yes     205
New1     No     7637
New2     Yes     206
New2     No     7364
")

model= glm(Pay ~ Website, weights=Weight, data=Data, 
           family=binomial(link="logit"))

library(car)

Anova(model, test="Wald")

   ### Analysis of Deviance Table (Type II tests)
   ### 
   ###         Df  Chisq Pr(>Chisq)  
   ### Website  2 5.9893    0.05006 

If we wanted to compare pairwise among the individual websites, we could use estimated marginal means. If we translate these marginal means from the analysis back to their original, "response", scale, we will get estimates for the proportions and their confidence intervals. Note that in this case, everything is quite close to those given by @BruceET .

library(emmeans)

marginal = emmeans(model, ~ Website, type="response")

marginal

   ### Website   prob      SE  df asymp.LCL asymp.UCL
   ### New1    0.0261 0.00180 Inf    0.0228    0.0299
   ### New2    0.0272 0.00187 Inf    0.0238    0.0311
   ### Old     0.0226 0.00108 Inf    0.0206    0.0248
   ###
   ### Confidence level used: 0.95 
   ### Intervals are back-transformed from the logit scale

Now, pairwise comparisons among individual websites with a Tukey adjustment for multiple comparisons.

pairs(marginal)

   ### contrast    odds.ratio     SE  df z.ratio p.value
   ### New1 / New2       0.96 0.0960 Inf -0.413  0.9104 
   ### New1 / Old        1.16 0.0999 Inf  1.722  0.1969 
   ### New2 / Old        1.21 0.1040 Inf  2.204  0.0705 
   ###
   ### P  value adjustment: tukey method for comparing a family of 3 estimates 
   ### Tests are performed on the log odds ratio scale 

If we wanted to look at these comparisons without the adjustment.

pairs(marginal, adjust="none")

   ### contrast    odds.ratio     SE  df z.ratio p.value
   ### New1 / New2       0.96 0.0960 Inf -0.413  0.6798 
   ### New1 / Old        1.16 0.0999 Inf  1.722  0.0850 
   ### New2 / Old        1.21 0.1040 Inf  2.204  0.0275

CHI-SQUARE TEST OF ASSOCIATION

Another approach is to arrange the counts in a contingency and use a chi-square test of association. Here the p value is just less than 0.05, so quite close to the previous analysis.

Input =("
Website  Pay   Nonpay
Old      425    18364
New1     205     7637
New2     206     7364
")

Matrix = as.matrix(read.table(textConnection(Input),
                   header=TRUE,
                   row.names=1))

Test = chisq.test(Matrix)

Test

   ### Pearson's Chi-squared test
   ###
   ### X-squared = 6.0033, df = 2, p-value = 0.0497

You might find some different different methods for post-hoc testing of chi-square test of association. Here, I'll suggest one approach: examining the standardized residuals. A standardized residual > 1.96 or < -1.96 corresponds to a p value of 0.05, and suggests that that cell is "interesting" relative to the expected values. Here, the cells in the Old row meet this criteria, suggesting that the cells in the Old row stand out "significantly" in the table.

Test$stdres

   ###            Pay    Nonpay
   ### Old  -2.412033  2.412033
   ### New1  1.108878 -1.108878
   ### New2  1.767987 -1.767987
$\endgroup$
0
$\begingroup$

I'd go for a one-sided paired difference test to test whether the difference in ratios is statisticially different from zero. In your case, you would make use of the t-distribution instead of the normal one. However, they are asymptotically the same. Then you could test two differences in ratios ($r$): $r_{new_1} - r_{old} > 0$ and $r_{new_2} - r_{old} > 0$ for both new websites.

$\endgroup$
1
  • $\begingroup$ Sorry, don't see how this test is applicable. $\endgroup$
    – BruceET
    Nov 7, 2019 at 10:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.