12
$\begingroup$

We are testing an e-mailing marketing campaign. On our initial test, we sent out two different e-mail types and had a third control group that did not receive an e-mail. Now we are getting back "results" as proportion of users who returned to our app. Here are the results:

  Group | received e-mail | returned | %-returned
  -----------------------------------------------
  A     | 16,895          | 934      | 5.53%
  B     | 17,530          | 717      | 4.09%
  C     | 42408           | 1618     | 3.82%

It looks like Group A may actually be better than B and C, but what is the proper test to show this?

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ Recall that the standard deviation of the proportion in a Binomial experiment involving $n$ independent results with success probability $p$ is $\sqrt{p(1-p)/n}$. Plugging in the estimates of $p$ gives standard errors of $0.18$%, $0.15$%, and $0.09$%, respectively. (These values can easily be estimated just by looking at the data: no computer is needed.) Because the difference $5.53$% - $\max(4.09, 3.82)$% = $1.44$% equals almost ten of any of those standard errors, the result is glaringly obvious that A has a greater return rate than B or C and no formal testing is needed. $\endgroup$
    – whuber ♦
    Commented Apr 16, 2013 at 21:35
  • $\begingroup$ @whuber Just a quick follow-up question. In this case, the normal approximation can be made, but what if the %'s were even smaller, say < 1%. What test would make sense in that case? $\endgroup$
    – thecity2
    Commented Apr 16, 2013 at 22:17
  • 1
    $\begingroup$ Good question. What matters are not the percentages but the actual counts. Don't worry until those counts (or their complements--the numbers not received) are around 30 or less (depending on how clear the results are, sometimes even counts of 5 can be ok). When percents and counts are both low, the Poisson approximation is great and you should consider logistic regression, as @gung recommends. That's a good general approach, too. $\endgroup$
    – whuber ♦
    Commented Apr 16, 2013 at 22:34

2 Answers 2

Reset to default
4
$\begingroup$

In a table like this you can partition the G-statistic produced by a G-test, rather than calculating the ORs or by running a logistic regression. Although you have to decide how you're going to partition it. Here the G-statistic, which is similar to Pearson's X^2 and also follows a X^2 distribution, is:

G = 2 * sum(OBS * ln(OBS/EXP)).

You first calculate that for the overall table, in this case: G = 76.42, on 2 df, which is highly significant (p < 0.0001). That is to say that return rate depends on the group (A, B, or C).

Then, because you have 2 df, you can perform two smaller 1 df (2x2) G-tests. After performing the first one, however, you have to collapse the rows of the two levels used in the first test, and then use those values to test them against the third level. Here, let's say you test B against C first.

Obs   Rec    Ret    Total
B   17530    717    18247
C   42408   1618    44026

Exp     Rec    Ret  Total
B   17562.8  684.2  18247
C   42375.2 1650.8  44026

This produces a G-stat of 2.29 on 1 df, which is not significant (p = 0.1300). Then make a new table, combining rows B and C. Now test A against B+C.

Obs   Rec    Ret    Total
A   16895    934    17829
B+C 59938   2335    62273

Exp     Rec    Ret  Total
A   17101.4  727.6  17829
B+C 59731.6 2541.4  62273

This produces a G-stat of 74.13, on 1 df, which is also highly significant (p < 0.0001).

You can check your work by adding the two smaller test statistics, which should equal the larger test statistic. It does: 2.29 + 74.13 = 76.42

The story here is that your B and C groups are not significantly different, but that group A has a higher return rate than B and C combined.

Hope that helps!

You could also have partitioned the G-stat differently by comparing A to B first, then C to A+B, or by comparing A to C, then B to A+C. Additionally, you can expand this to 4 or more groups, but after each test you have to collapse the two rows that you just tested, with a maximum number of tests equal to the df in your original table. There are other ways to partition with more complicated tables. Agresti's book, "Categorical Data Analysis", should have the details. Specifically, his chapter on inference for two-way contingency tables.

Improve this answer
$\endgroup$
1
$\begingroup$

I would simply calculate odds (or risk) ratios between group A and B, between B and C, and between A and C and see if they statistically different. I don't see a reason to do a "omnibus" proportions test in this case since you only have three groups. Three chi-square tests could do the trick as well.

As some of the individuals have outlined in the comments below, and logistic regression with planned contrasts would work well too.

Improve this answer
$\endgroup$
5
  • 3
    $\begingroup$ There is a potential multiple comparisons issue here. Why not just do a logistic regression with 2 dummy codes for B & C? $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 16, 2013 at 20:57
  • 1
    $\begingroup$ Yes, sure, but the issue of multiple comparisons is pretty minimal when you have three comparisons. $\endgroup$
    – Behacad
    Commented Apr 16, 2013 at 21:33
  • $\begingroup$ @gung makes some good points. Logistic regression would be the simplest approach -- and if one were doing a chi-squared test approach, then you'd almost definitely start off with an omnibus test (of the 3x2 table contingency table) before doing two-group comparisons (although this would correspond to the "overall" significance of the logistic regression model fit in this instance.) $\endgroup$
    – James Stanley
    Commented Apr 16, 2013 at 21:36
  • 1
    $\begingroup$ Using this suggestion (and Wikipedia), I found that the 95% CI for the log odds of A/B and A/C did not overlap with 0, and that the log odds ratio for B/C did overlap with 0. Does that signify that A is significantly different from B & C? $\endgroup$
    – thecity2
    Commented Apr 16, 2013 at 21:42
  • $\begingroup$ @Behacad, you're right that w/ only 3 comparisons, multiple comparisons issues would be less intense, but I would still start w/ a LR model. Ideally, that would be followed up with planned comparisons. $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 16, 2013 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.