Statistical Significance on shops performances with a marketing promotion

Question

I have data regarding the performances of shops on a certain time period: in this data some shops had an active marketing promotion, and some didn't. I would like to understand whether the shops with the promotion improved their performances more than the other group.

I have formulated the problem in the following terms: how many shops improved their performances (wrt the previous time period) among the two groups? In this case, my results are

Group	Total # Shops	# Shops with improved performances
No Promo	37907	23762
With Promo	1007	801

Running a significance test, data suggests that the difference is significant (p-value < 0.05), so I would have concluded that shops with active promo are more likely to improve their performances.

I would like now to estimate how much the improvement is, so if there's a difference in the mean growth between the two groups. In this case, data are:

Group	Total # Shops	Mean Growth	Std
No Promo	37907	0.45	0.8896
With Promo	1007	0.83	1.0367

At this point, do I have to run a test on the difference in growth mean to understand if they are significant? If so, which kind of test should be used?

Answer 1

It seems you're in need of the chi-square test, which tests for variable independency. If the test is significant (let's consider p < .05), your variables are related to each other.

I've created your example in R using the following code:

# creating promo
promo <- c(rep('no', 37907), rep(c('yes'), 1007)) # creating promo

# creating improvement
improved <- c(rep('yes', 23762), rep('no', 14145), rep('yes', 801), rep('no', 206)) # creating improvement

# creating dataframe object
df <- data.frame(promo = promo, improved = improved)

I then read library(gmodels) and tested for independece using the function CrossTable()

x <- CrossTable(df$promo, df$improved, chisq = T, fisher = T)

The resulting table is the following:

Total Observations in Table:  38914 

 
             | df$improved 
df$promo |        no |       yes | Row Total | 
-------------|-----------|-----------|-----------|
          no |     14145 |     23762 |     37907 | 
             |     1.956 |     1.143 |           | 
             |     0.373 |     0.627 |     0.974 | 
             |     0.986 |     0.967 |           | 
             |     0.363 |     0.611 |           | 
-------------|-----------|-----------|-----------|
         yes |       206 |       801 |      1007 | 
             |    73.638 |    43.023 |           | 
             |     0.205 |     0.795 |     0.026 | 
             |     0.014 |     0.033 |           | 
             |     0.005 |     0.021 |           | 
-------------|-----------|-----------|-----------|
Column Total |     14351 |     24563 |     38914 | 
             |     0.369 |     0.631 |           | 
-------------|-----------|-----------|-----------|

The actual statistical results are:

Statistics for All Table Factors


Pearson's Chi-squared test 
------------------------------------------------------------
Chi^2 =  119.7606     d.f. =  1     p =  7.137585e-28 

Pearson's Chi-squared test with Yates' continuity correction 
------------------------------------------------------------
Chi^2 =  119.0375     d.f. =  1     p =  1.027699e-27 

 
Fisher's Exact Test for Count Data
------------------------------------------------------------
Sample estimate odds ratio:  2.314598

Note that p < .05, it's even below .001. That being said, p is not an effect size measure, and that's what you're trying to know, as I understand.

Note the Sample estimate odds ratio: 2.314598 in the final line of the output. That's your effect size. What that number means is that the variable promo increases the odds of improved performances of a store by 2.31 times.