I have got data from a randomized control trial experiment with outcome whether the individual made a purchase or not.

How do I estimate the average treatment effect if my outcome is binary? In this case is average treatment effect the difference in the percentage of users who purchase a product between the two groups?

In other words, some kind of contingency table? Related question, How would you calculate if the results are statistically significant?

calculate p-value for binomial distribution?

  • $\begingroup$ Could you describe a bit more your dataset ? For instance, do you have a table with a couple of varying parameters and the binary outcome, or just two columns (experiment label, e.g. exp_A, exp_B, ..., and the outcome) ? $\endgroup$ – AshOfFire May 2 '18 at 8:17
  • $\begingroup$ I have got a set of control variables. The groups ( control and treatment) each observation from the experiment belongs to and whether or not the user associated with the observation made a purchase. $\endgroup$ – kms May 2 '18 at 13:31

Assuming that you have a dataset like this.

ctrl1 ctrl2 ctrl3       grp outcome
    2     4     a   control       0
    3     5     a   control       1
    3     5     a   control       0
    2    11     b treatment       1
    3    11     c treatment       1

Where ctrl designs your list of controls, grp the group associated to the test (control or treatment) and outcome the user final decision.

You can calculate a few statistics, even with binary output.

  • Contingency table and $\chi^2$ test to evaluate the independence between your variable and your outcome.
  • Correlation between the set of controls and the outcome. Although you may need to transform your dataset if you are handling categorical variables, this measure will highlight potential links between your variables and their influence on your output.

Since you are interested on the effect of your treatment, you can start with the $\chi^2$ test (aka Chi-Squared test). After transforming your data into a contingency table, you can use this test to verify if your outcome is independent from your variable.

grp         0  1
  control   80 28
  treatment 12 93

With this contingency table, you have one degree of freedom, and with a $\chi^2 = 82.619$, you can reject the hypothesis of independence with sufficient probability, which means that the treatment does have an effect on the outcome.

Correlation may come in a second step to seek for potential links between your list of controls and the outcome.

  • For numerical variables, you can assume the correlation will return directly positive or negative effects in the increase/decrease of the control.
  • For categorical variables, you may need to transform your control parameter into a dummy boolean for each value it may take. You can then either calculate the resulting correlation, or again use a contingency table.

You may be interested in reading the details of Chi-Squared Test, especially the Yates correction for binomial outputs.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.