Average treatment effect for binary outcome

Question

I have got data from a randomized control trial experiment with outcome whether the individual made a purchase or not.

How do I estimate the average treatment effect if my outcome is binary? In this case is average treatment effect the difference in the percentage of users who purchase a product between the two groups?

In other words, some kind of contingency table? Related question, How would you calculate if the results are statistically significant?

calculate p-value for binomial distribution?

Answer 1

Assuming that you have a dataset like this.

ctrl1 ctrl2 ctrl3       grp outcome
    2     4     a   control       0
    3     5     a   control       1
    3     5     a   control       0
    ...
    2    11     b treatment       1
    3    11     c treatment       1

Where ctrl designs your list of controls, grp the group associated to the test (control or treatment) and outcome the user final decision.

You can calculate a few statistics, even with binary output.

• Contingency table and $\chi^2$ test to evaluate the independence between your variable and your outcome.
• Correlation between the set of controls and the outcome. Although you may need to transform your dataset if you are handling categorical variables, this measure will highlight potential links between your variables and their influence on your output.

Since you are interested on the effect of your treatment, you can start with the $\chi^2$ test (aka Chi-Squared test). After transforming your data into a contingency table, you can use this test to verify if your outcome is independent from your variable.

            outcome
grp         0  1
  control   80 28
  treatment 12 93

With this contingency table, you have one degree of freedom, and with a $\chi^2 = 82.619$, you can reject the hypothesis of independence with sufficient probability, which means that the treatment does have an effect on the outcome.

Correlation may come in a second step to seek for potential links between your list of controls and the outcome.

• For numerical variables, you can assume the correlation will return directly positive or negative effects in the increase/decrease of the control.
• For categorical variables, you may need to transform your control parameter into a dummy boolean for each value it may take. You can then either calculate the resulting correlation, or again use a contingency table.

You may be interested in reading the details of Chi-Squared Test, especially the Yates correction for binomial outputs.