0
$\begingroup$

I'm trying to find the appropriate significance test to run for my given scenario. I seem to find conflicting advice and would appreciate any advice.

I'm trying to measure the impact of some media on the % of customers who have purchased a product (measured before and after the media was shown).

The media is not targeted so technically anyone could have seen it, we are collecting "Exposure" information via a retrospective survey (i.e. we've asked customers whether they have seen media and 1 = Yes, 0 = No).

We have the situation in the table below (I do have access to actual sample sizes but not shown here):

![enter image description here

The Test group proportion has increased 3.06% and the Control group proportion has increased 1.07%.

I want to know whether the increase the Test group saw was significantly higher than the Control group.

I appreciate that the design of the test may be somewhat flawed, for example in the way we are identifying Test & Control but I unfortunately don't have much control over this. My thinking was if we could assume that the % Purchased for Test & Control was the same, we could just run a standard Z test on the absolute % increase, however I don't think we are safe to make this assumption and as such I'm a little stuck.

$\endgroup$
0
$\begingroup$

It is important to look at counts and sample sizes, not just percentages or proportions:

If you have 891 subjects with counts as shown in the matrix DTA below, then a chi-squared test of homogeneity finds no significant difference at the 5% level in the percentage of purchases between Test and Control subjects: P-value above 5%.

DTA = matrix(c(214,244,211,222), byrow=T, nrow=2)
sum(DTA)
[1] 891
DTA
     [,1] [,2]
[1,]  214  244
[2,]  211  222

chisq.test(DTA, cor=F)  # Without Yates' continuity correction

        Pearson's Chi-squared test

data:  DTA
X-squared = 0.35863, df = 1, p-value = 0.5493

However, if you have 8909 subjects with counts about ten times as large, then there is a significant difference at the 5% level.

DTA = matrix(c(2135,2441,2116,2217), byrow=T, nrow=2)
sum(DTA)
[1] 8909
chisq.test(DTA, cor=F)

        Pearson's Chi-squared test

data:  DTA
X-squared = 4.2321, df = 1, p-value = 0.03967
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.