The data I want to analyze consist of set of votes similar to the voting system here on stackexchange. Votes are binary, i.e, items can receive up- or down-votes. The data have been gathered in an A/B controlled experiment.

I want to compare the control group to the treatment group according to some gold standard. That is, for some items I know what the "correct" vote (according to the gold standard) should have been. However, the number of votes per item varies and can be different between control and treatment group. My hypothesis is, that the treatment reduces the voting "error". But I am not sure, which statistical test to use.


1 Answer 1


If you know what the correct vote should be, you could start with a simple chi-square or KS test, testing for the number of votes misclassified for each group (treatment and control) as compared to the gold standard.

-Ralph Winters


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.