2
$\begingroup$

I have two groups of people. One group were shown an ad and the other were not. I know in aggregate that the treatment group spent \$799 and the control group spent \$412, so overall it seems people responded. Both groups had 1000 people in them.

I am not sure how to translate this to verify whether this is statistically significant. I was going to do a chi squared test with treatment group observed values and use the expected value of the average spend per customer in the control group (\$0.412).

Is this the correct way to choose an expected value?

$\endgroup$
  • 1
    $\begingroup$ More likely to be a t test, but watch for unequal variances and non-normality. Can you plot the data? Were there lots of zeros? $\endgroup$ – Nick Cox Jun 4 at 14:31
  • $\begingroup$ @NickCox 950/1000 of the treatment group have 0-values and 971/1000 of the control group have 0-values $\endgroup$ – J. Stott Jun 4 at 14:46
  • $\begingroup$ You'd be stretching the assumptions of a t test with such data, but you might get away with it with such a large pair of samples. You could cross-check with a different procedure. All you need to do is list the 50 and 29 non-zero values and people can play with the data. $\endgroup$ – Nick Cox Jun 4 at 14:49
  • 2
    $\begingroup$ A chi-square test (of the kind you're proposing in your question) is for counts, not dollars. $\endgroup$ – Glen_b Jun 5 at 2:59
  • 1
    $\begingroup$ I was trying to be positive, but as @Glen_b points out gently, using mean purchase as it were expected frequency would be quite wrong. $\endgroup$ – Nick Cox Jun 5 at 8:07
4
$\begingroup$

Comment: There is one sense in which seeing the ad seems to have prompted a greater response. That is the response rate (rather than the dollar amount spent).

There are various tests of $H_0: p_1 = p_2$ vs. $H_a: p_1 > p_2,$ where the $p_i$ are the respective response rates.

Output from Minitab for two such tests is shown below.

Test and CI for Two Proportions 

Sample   X     N  Sample p
1       50  1000  0.050000
2       29  1000  0.029000

Difference = p (1) - p (2)
Estimate for difference:  0.021
95% lower bound for difference:  0.00669270
Test for difference = 0 (vs > 0):  
   Z = 2.41  P-Value = 0.008

Fisher’s exact test: P-Value = 0.011

The first test (P-value 0.021) uses a normal approximation of the binomial proportions, which should give reasonably accurate results for such large samples. Fisher's exact test (P-value 0.011) uses a hypergeometric distribution. Both tests are significant.

If looking at response rates is of interest to you, you can find particulars in an elementary applied statistics text or online.

As @NickCox suggests, we would have to know the $(50+29 = 79)$ individual dollar amounts in order confidently to explore two-sample tests for amount spent. However, it seems each purchase in each group averages around $15, so looking at response rates might tell you what you really want to know about the effect of exposure to the ad.


Note: Just as an experiment, I simulated a dataset assuming that the 50 nonzero sales in Group 1 are distributed $\mathsf{Norm}(\mu=16, \sigma=3)$ and that the 29 nonzero sales in Group 2 are distributed $\mathsf{Norm}(\mu=14, \sigma=3).$ Dollar amounts were rounded to integers:

table(x1)
x1
  0  10  11  12  13  14  15  16  17  18  19  20  21  22 
950   2   1   7   6   8   5   4   5   2   2   5   1   2 

table(x2)
x2
  0   8   9  10  11  12  13  14  15  16  18  21  22 
971   1   2   1   3   4   2   3   4   3   4   1   1 

A Welch two-sample t test in R gave P-value 0.0033, as follows:

        Welch Two Sample t-test

data:  x by g
t = 2.7117, df = 1803.6, p-value = 0.003379
alternative hypothesis: 
  true difference in means is greater than 0
95 percent confidence interval:
  0.1415163       Inf
sample estimates:
mean in group 1 mean in group 2 
          0.768           0.408 

In spite of all the zeros and additional ties that result from rounding dollars to integers, a one-sided, two-sample Wilcoxon (rank sum) test in R gave P-value 0.007. with no error messages.

wilcox.test(x ~ g, alt="g")$p.val
[1] 0.00725217

A one-sided permutation test with the pooled 2-sample t statistic as metric (but not assuming normality) gave P-value about 0.003.

Unless your non-zero dollar values are much different from my simulated ones, I do not expect a problem finding a valid two-sample test to compare dollar amounts.

$\endgroup$
  • 1
    $\begingroup$ @NickCox. Thanks for edits. $\endgroup$ – BruceET Jun 5 at 8:01
  • $\begingroup$ You're welcome. It's a very nice answer. I would be proud to have written it. $\endgroup$ – Nick Cox Jun 5 at 10:23
  • $\begingroup$ Thank you for this! I will take a look at your reply this evening when I have a chance to digest it. I really appreciate the effort you've put into this. $\endgroup$ – J. Stott Jun 5 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.