We run a survey and want to run a controlled test on a sample of the data. The measure we are trying to improve is response rate.

Our research question is:

“does sending out more invites to participate, improve the response rate for the treatment group versus the control group that receive less invites?”

The dependent variable is whether the person takes the survey, and this is a simple categorical variable: respond / do not respond.

However, our measure for this variable is continuous, in that we are measuring the response rate.

We are measuring for both the treatment group and the control group: number of responses/population

Our total population is large 30,000 people split into a 3,000 treatment group and 27,000 people in the control group. Therefore, we think the large population is likely to lead to a statistically significant result even if the difference between the groups could be due to natural variance.

I’m not sure what statistical tests are bests to run on the data? Most tests require a mean and standard deviation, but I’m not sure we can calculate these effectively with a categorical variable. Or whether I should be treating the data as continuous?

I’ve looked at odds ratio test and this seems to be the best fit I can see in terms of comparing the response rates. I don’t know if using Cohens d test might be a better option?


1 Answer 1


Even if you are most interested in rates, if you have the number of respondents and non-respondents in both groups, the most common test here would be a simple $\chi^2$ test on contingency tables.

And good catch on understanding that even trivial differences can be statistically significant with large sample sizes. Best to report the actual rates with confidence intervals, possibly also ORs with CIs.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.