How do i calculate if the difference between two advertisement campaigns is significant?

Question

My brother has asked me if I can evaluate if his advertisement campaigns yield significantly different results. It was a couple of years ago I dealt with these kinds of problems, so would appreciate some feedback.

For the sake of simplicity, we'll try to see if there's a difference between his normal campaign and his most successful one.

Normal campaign	Most successful
Leads(n)=106	Leads(n)=9
Successes=8	Successes=8
Success ratio(p)=0.075	success ratio=0.44
Variance =$p(1-p)=0.075*(1-0.075)/106=0.00066$	variance =$p(1-p)/n=0.44(1-0.44)/9=0.0273$

My notes are a bit unclear, but I think that I need to form a pooled variance by adding both variances together, and then take the square root in order to get the standard deviation. In that case this would yield:

$$0.0273+0.0006=0.0279 0.0279^{(1/2)}=0.167$$

We should then (I think) calculate the difference in success ratio and divide this value with the pooled standard deviation:

$$0.44-0.075=0.365 0.365/0.167=2.18$$

That is to say we have a z score of 2.18, which means the result is significant on a 95 percent confidence level.

Question 1:

Are these steps valid, or did I make a mistake somewhere?

Question 2:

Is the method valid for these kinds of values (I've read that it's best when $np\geq 10$ for instance) and if not, is there a method that's more accurate?

Answer 1

You propose a test to see it two binomial proportions are the same (null hypothesis) or whether one is larger than the other (one-sided alternative).

Minitab, does such a test, using a normal approximation, as follows:

Test and CI for Two Proportions 

Sample  X    N  Sample p
1       8  106  0.075472
2       8    9  0.888889

Difference = p (1) - p (2)
Estimate for difference:  -0.813417
95% upper bound for difference:  -0.636015
Test for difference = 0 (vs < 0):  
  Z = -7.54  P-Value = 0.000

* NOTE * The normal approximation may be inaccurate for small samples.

Fisher’s exact test: P-Value = 0.000

The P-value is quite small, indicating that the second method ('campaign') is significantly better at any reasonable level of significance. However, because the sample size for the second method is so small, the normal approximation may not give an exactly correct P-value.

Fisher's exact test is often used when sample sizes are too small for a normal approximation to give useful results. It also gives a tiny P-value. You can google for discussions of Fisher's exact test, if you are not familiar with it.

Note: I have one serious reservation about this inference. If your brother chose the one "most successful" campaign out of many, just on the basis of the high number of successes, then I wonder about the comparison. If we pick the best out of dozens of his campaigns (perhaps with widely varying results with low numbers of trials), then a low P-value would not be surprising (or convincing).