Assuming that you have a dataset like this.
ctrl1 ctrl2 ctrl3 grp outcome
2 4 a control 0
3 5 a control 1
3 5 a control 0
2 11 b treatment 1
3 11 c treatment 1
ctrl designs your list of controls,
grp the group associated to the test (control or treatment) and
outcome the user final decision.
You can calculate a few statistics, even with binary output.
- Contingency table and $\chi^2$ test to evaluate the independence between your variable and your outcome.
- Correlation between the set of controls and the outcome. Although you may need to transform your dataset if you are handling categorical variables, this measure will highlight potential links between your variables and their influence on your output.
Since you are interested on the effect of your treatment, you can start with the $\chi^2$ test (aka Chi-Squared test). After transforming your data into a contingency table, you can use this test to verify if your outcome is independent from your variable.
grp 0 1
control 80 28
treatment 12 93
With this contingency table, you have one degree of freedom, and with a $\chi^2 = 82.619$, you can reject the hypothesis of independence with sufficient probability, which means that the treatment does have an effect on the outcome.
Correlation may come in a second step to seek for potential links between your list of controls and the outcome.
- For numerical variables, you can assume the correlation will return directly positive or negative effects in the increase/decrease of the control.
- For categorical variables, you may need to transform your control parameter into a dummy boolean for each value it may take. You can then either calculate the resulting correlation, or again use a contingency table.
You may be interested in reading the details of Chi-Squared Test, especially the Yates correction for binomial outputs.