Correct use of Chi-square and hypothesis testing?

Question

I have a dataset which contains lots of data on customers and their actions within their journey through our business, one of these actions is how many of our events they have attended, another action is whether they have made a purchase from us. I am testing the hypothesis that attending an event makes someone more likely to make a purchase.

For this problem I used the following hypotheses: H_0: There is no difference in the percentage of people to make a purchase between the two groups. H_1: The percentage of people to make a purchase for those that have attended an event is higher than for those that haven't.

To test these hypothesis I produced a contingency table as shown below:

	Made a purchase	Didn't make a purchase
Attended an event	190	1350
Didn't attend an event	983	15588

And then used the scipy.stats function 'chi2_contingency()' to do a test of independence. I thought it was right to either use the Chi-Squared test or Fisher's exact test but since the sample size is quite large then Chi-Squared was more suitable:

chistat, pvalue, dof, ex = chi2_contingency(cont_df)
print(chistat, pvalue)

This then gave the output:

(94.39478042474279, 2.5847291047881143e-22)

Which I thought showed that when using significance level of 0.05 there was a statistically signifcant difference between the probability to make a trade between the group that attended an event and the group which didn't, hence rejecting the null hypothesis.

This is my first time applying hypothesis testing to a real world problem and so would appreciate if anyone could point out any flaws in my process/understanding. My main issue with it lies in the fact that people who are going to attend an event are presumably already more likely to make a purchase, so this doesn't necessarily prove any causality between the two variables. As well as this, would it not be better to have a control group which didn't even have the option of attending an event i.e. did receive an email from us advertising the event. Then performing a test using one group as those that were aware of the events, and one of those that weren't and see if there is a statistically significant difference in purchase conversion between those two groups?

Answer 1

All of your concerns are valid and well articulated. The chi-square test simply provides the weight of the evidence that there is an association between attendance and purchase. Any causality is an extra layer of interpretation that in many instances is unverifiable. There are many causal hypotheses one could entertain.

Randomly assigning people to either attend or not attend the event would allow you to make a stronger causal inference argument as to the effect attendance has on making a purchase. This is because through randomization each subject in the study would be equally likely to attend the event (the distribution of subject features is, in the long run, the same between the two groups). Any difference in purchasing behavior could then be attributed to attendance alone.

This doesn't mean that all observational studies are useless. It just means that you have to preface every statement with, "Assuming no unmeasured confounders...[insert causal statement here]," or state that the findings show an association without speculating as to the causal nature. I generally do both, starting with the latter and finishing with the former.

Addendum: If you have an imbalance in prognostic factors between those who attend and those who do not attend, then for causal inference you have at least a couple of options: 1) adjust the outcome model and interpret the effect of attendance in various subgroups (main effect, interactions). For causal inference you may consider drawing a causal diagram to understand how the covariates should enter the model. 2) Re-weight the observations using IPTWs. This balances the measured prognostic covariates between treatment groups to make the sample appear as though it is the result of a randomized trial. The outcome model can be analyzed using only a main effect for attendance. Causal inference on the effect of attendance is for the entire population. With either approach the causal interpretation of the results will still need to be prefaced with, "Assuming no unmeasured confounders..."

Answer 2

An alternative version of your chi-squared test is the prop.test procedure in R. You have the proportion of purchasers $\hat p_a = 190/1540 = 0.1234$ among those who attended events and the proportion $\hat p_n = 893/16571 = 0.0593$ among those who did not attend events. The question is whether $\hat p_a$ and $\hat p_n$ are significantly different at the 5% level.

The test is shown below. (I declined continuity correction on account of the relatively large sample sizes.)

Sometimes the output from prop.test is easier to interpret than output from the (equivalent) chi-squared test.

prop.test(c(190,983), c(1540, 16571), cor=F)

        2-sample test for equality 
        of proportions without continuity correction

data:  c(190, 983) out of c(1540, 16571)
X-squared = 95.449, df = 1, p-value < 2.2e-16
alternative hypothesis: two.sided
95 percent confidence interval:
 0.04724175 0.08087049
sample estimates:
   prop 1    prop 2 
0.1233766 0.0593205 

I agree with @GeoffreyJohnson's answer (+1) about a design for making a stronger case. However, the chi-squared test can also be viewed as a test whether the two categorical variables are independent. So properly stated, as you have done, your conclusions from the current data are valid. You have shown a statistical association, not causation.

For comparison, I show output from the chisq.test procedure in R (without Yates' correction):

TAB = rbind(c(190, 983), c(1350, 15588))
TAB
     [,1]  [,2]
[1,]  190   983
[2,] 1350 15588

chisq.test(TAB, cor=F)

        Pearson's Chi-squared test

data:  TAB
X-squared = 95.449, df = 1, p-value < 2.2e-16

Answer 3

On a shallow level, the analysis you've done is good enough to show to the business. This is what matters.

Now, "is our marketing good and people buy products because of these events?" is quite different from "should we keep conducting these events, and if so, how many?". Maybe events make people feel better about the company, not the product. Maybe with 10 more events you would sell 3x as much stuff. Maybe you would sell as much with 5x fewer events. Your analysis does show a correlation between events and purchases but does not answer all the questions a business might imply being the same, but posing a totally different and, likely, more complex problem.

That makes it your priority to focus on communication and making sure "yes, events are probably good for sales" does not translate to some uncontrollable and immeasurable actions.

On the topic of the problem itself - one thing to do next is to classify your clientele. Some are repeated customers, some buy products first and then attend events to get support on them, some are indeed drawn in by the marketing... Understanding this structure better should give you tools to deal with "what should we do" kinds of problems. More broadly, consider some common customer engagement metrics: you mention some of them in your post already, but this is a well-developed area - doing your research first is (nearly) always a good idea ;)