What is the correct test for differences of proportions with this design?

Question

I have a series of questions with binary choices, which are coded as 1 and 0. I created a percentage by adding up the "1" responses and dividing by the total number of questions. For example, for one respondent, if they gave a "1" response for 6 out of 10 questions, their rate would be 60%.

The design of the study is so that there are two different groups of respondents. One group got series A questions, and the other group got series B questions. Then, I calculated the mean percentage for each group. Series A is the control, while series B is the treatment. What is the correct statistical test to see whether there is a significant difference between these mean percentages?

Answer 1

I disagree with the suggestion on contingency tables and chi-square testing as it won't help you test the difference between means. You've used your 10 questions to create a rating scale, and with that scale, means do apply.

People do differ as to how finely grained a variable must be before one can legitimately test means. You'll see many disagreements over variables measured on a 1-5 scale. But with a 0-10 scale, very few would object to the use of a means test.

You used the "proportions" tag. It's true that proportions are involved to the extent that each person gives a "1" response to some proportion of the 10 questions. But once you go beyond the 0's and 1's and deal with the scale scores (what you call "ratings"), you are interested in the comparing the mean for the entire group, A vs. B. You no longer need think about proportions at that point, just the mean of each set of scores. So a test of the difference between means (z-test or t-test for independent samples) is what you're after.

To illustrate why "proportion" no longer applies, suppose one group's scores are {.6, .3, .6, .6, .4}. You could transform all such scores to {6, 3, 6, 6, 4} or even {60, 30, 60, 60, 40} without affecting a t-test's results.