I'm trying to find the appropriate significance test to run for my given scenario. I seem to find conflicting advice and would appreciate any advice.
I'm trying to measure the impact of some media on the % of customers who have purchased a product (measured before and after the media was shown).
The media is not targeted so technically anyone could have seen it, we are collecting "Exposure" information via a retrospective survey (i.e. we've asked customers whether they have seen media and 1 = Yes, 0 = No).
We have the situation in the table below (I do have access to actual sample sizes but not shown here):
The Test group proportion has increased 3.06% and the Control group proportion has increased 1.07%.
I want to know whether the increase the Test group saw was significantly higher than the Control group.
I appreciate that the design of the test may be somewhat flawed, for example in the way we are identifying Test & Control but I unfortunately don't have much control over this. My thinking was if we could assume that the % Purchased for Test & Control was the same, we could just run a standard Z test on the absolute % increase, however I don't think we are safe to make this assumption and as such I'm a little stuck.