To measure the effectiveness of a marketing campaign, I have divided my customers in two groups. The experiment group, which is exposed to my campaign and the control group which is not exposed to the campaign. Experiment Group, Sample size is 10,000 and Control group sample size is 1000. My outcome is the buying response from the customer. So, the customer can buy irrespective of whether he/she is exposed to the campaign. I want to analyze the effectiveness of my campaign and find out whether the proportion of customers who bought, differs in the two groups - experiment and control. Shall I perform a t-test? My data is actually binary. The customer either buys or doesn't. Also, how should I determine my sample sizes for the control and experiment groups before conducting the exercise.
1 Answer
"Shall I perform a t-test?" No. A t-test assumes that the dependent variable is continuous in nature. As you mention, your dependent variable is binary in nature, making a t-test inappropriate. See e.g. Wikipedia Student's t-test assumptions.
Instead you can use tests designed for counts, such as a test of equal proportions or chi-square test of association. An example is given below with code for R.
The question of sample size may need to be a separate question. There are methods to calculate the sample size needed for a chi-square test. (See for example the section on power analysis at the bottom of Handbook of Biological Statistics: Chi-square. However, in your case there is a secondary consideration in that you probably won't want to divide your sample equally, between an established ad campaign known to be effective and an experimental one.
Researching A/B testing in general may be helpful.
The following examples assumes that in the Control group 200 out of 10000 purchased, and in the Campaign group 30 out of 1000 purchased.
Resources:
### http://rcompanion.org/handbook/H_10.html
### http://rcompanion.org/handbook/H_04.html
Install packages:
if(!require(psych)){install.packages("psych")}
if(!require(DescTools)){install.packages("DescTools")}
if(!require(ggplot2)){install.packages("ggplot2")}
Test of equal proportions. Results show that the campaign was more effective. Proportion purchasing was 2 % for Control and 3 % for Campaign. This was a significant difference (p = 0.046).
Bought = c(30, 200)
N = c(1000, 10000)
prop.test(Bought, N)
# X-squared = 3.9656, df = 1, p-value = 0.04644
#
# sample estimates:
# prop 1 prop 2
# 0.030 0.020
Results will be similar with a chi-square test of association. Note that here we need to use the Bought and NotBought, not Bought and Total N.
Input =("
Method Bought NotBought
Control 200 9800
Campaign 30 970
")
Matrix = as.matrix(read.table(textConnection(Input),
header=TRUE,
row.names=1))
Matrix
Determine the proportions in each row of the table.
prop.table(Matrix, margin=1)
# Bought NotBought
# Control 0.02 0.98
# Campaign 0.03 0.97
Chi-square test of association. The result here is significant, suggesting there is an association between the campaign method and the proportion of recipients purchasing (p = 0.046).
chisq.test(Matrix)
# X-squared = 3.9656, df = 1, p-value = 0.04644
It is important to look at the size of the effect.
One method is to look at the difference in the rates, in this case 2% to 3%. Is this a meaningful difference for the business, especially considering any additional costs of the campaign?
Another method is to use an effect-size statistic. In this case (a 2 x 2 contingency table), the phi statistic can be used. It ranges from 0 (no association) to 1 (perfect association), with a + or - depending on direction of the association. This is a standardized statistic for the degree of association in the table, so could be used to compare among other 2 x 2 tables.
library(psych)
phi(Matrix,
digits = 4)
# [1] -0.0201
It is an assumption of the chi-square and equal proportion tests that the "expected" values aren't too small. So, for example, if I were to change the counts of Bought to 20 and 3, the expected counts would be too low for this kind of test to be valid. However, Fisher's exact test could be used in this case since you are fixing the number of Control and Campaign counts.
chisq.test(Matrix)$expected
fisher.test(Matrix)
# p-value = 0.04749
As a final note, it is useful to look at the confidence intervals of the proportions. The code below gets a little complicated, but looking at the confidence interval for Campaign, it ranges down to almost 0.02, so we don't have much certainty that the proportion for Campaign is really that much different than for Control. If the sample sizes were more uniform, we might have more narrow confidence intervals.
library(DescTools)
Campaign = BinomCI(30, 1000,
conf.level = 0.95,
method = "clopper-pearson")
Control = BinomCI(200, 10000,
conf.level = 0.95,
method = "clopper-pearson")
Data=data.frame(Method = c("Campaign", "Control"),
Proportion.bought = c(NA, NA),
LowerCI= c(NA,NA),
UpperCI= c(NA,NA))
Data[1,2:4] = Campaign[1,1:3]
Data[2,2:4] = Control[1,1:3]
Data
# Method Proportion.bought LowerCI UpperCI
# 1 Campaign 0.03 0.02033049 0.04255140
# 2 Control 0.02 0.01734654 0.02293792
Plot the proportions and confidence intervals
library(ggplot2)
ggplot(Data,
aes(x = Method,
y = Proportion.bought)) +
geom_bar(stat = "identity",
color = "black",
fill = "gray50",
width = 0.7) +
geom_errorbar(aes(ymin = LowerCI,
ymax = UpperCI),
width = 0.2,
size = 0.7,
color = "black"
) +
theme(axis.title = element_text(face = "bold"),
axis.text = element_text(face = "bold")) +
ylab("Proportion purchased") +
xlab("Method")
