# Appropriate test for change in proportions of Test & Control, before & after exposure

I'm trying to find the appropriate significance test to run for my given scenario. I seem to find conflicting advice and would appreciate any advice.

I'm trying to measure the impact of some media on the % of customers who have purchased a product (measured before and after the media was shown).

The media is not targeted so technically anyone could have seen it, we are collecting "Exposure" information via a retrospective survey (i.e. we've asked customers whether they have seen media and 1 = Yes, 0 = No).

We have the situation in the table below (I do have access to actual sample sizes but not shown here):

The Test group proportion has increased 3.06% and the Control group proportion has increased 1.07%.

I want to know whether the increase the Test group saw was significantly higher than the Control group.

I appreciate that the design of the test may be somewhat flawed, for example in the way we are identifying Test & Control but I unfortunately don't have much control over this. My thinking was if we could assume that the % Purchased for Test & Control was the same, we could just run a standard Z test on the absolute % increase, however I don't think we are safe to make this assumption and as such I'm a little stuck.

It is important to look at counts and sample sizes, not just percentages or proportions:

If you have 891 subjects with counts as shown in the matrix DTA below, then a chi-squared test of homogeneity finds no significant difference at the 5% level in the percentage of purchases between Test and Control subjects: P-value above 5%.

DTA = matrix(c(214,244,211,222), byrow=T, nrow=2)
sum(DTA)
[1] 891
DTA
[,1] [,2]
[1,]  214  244
[2,]  211  222

chisq.test(DTA, cor=F)  # Without Yates' continuity correction

Pearson's Chi-squared test

data:  DTA
X-squared = 0.35863, df = 1, p-value = 0.5493


However, if you have 8909 subjects with counts about ten times as large, then there is a significant difference at the 5% level.

DTA = matrix(c(2135,2441,2116,2217), byrow=T, nrow=2)
sum(DTA)
[1] 8909
chisq.test(DTA, cor=F)

Pearson's Chi-squared test

data:  DTA
X-squared = 4.2321, df = 1, p-value = 0.03967