A survey question asks to tick all options that are relevant to the users (Check all that apply). Respondents are given 6 options and they can tick as many as they want.
I want to determine which are the top 3 options.
I can sort the options by frequency (see data below), but that does not tell me whether e.g. the third most frequent option is significantly more frequent than the fourth.
I want to test whether the difference in frequency is statistically significant between the 6 options and compare the different options (post hoc).

My best option is to calculate the confidence interval around each frequency and compare those, but I wonder whether there is a more rigorous test.

This is the data (I sorted it by descending frequency)
sample size n=150

  • Feature Frequency
  • qualifications 82%
  • work_experience 75%
  • case studies 53%
  • personal_brand 49%
  • awards 42%
  • career_goal 12%
  • $\begingroup$ Use Cochran's Q test with subsequent multiple comparisons. For your situation, I may recommend not all pairwise comparisons but the stepwise stepdown procedure which isolates subsets of items between which the difference is statistically most significant. Also to note, the Q test is just the Friedman test in case of binary data (your data are 6 binary variables). $\endgroup$ – ttnphns Sep 24 at 7:44
  • $\begingroup$ Thanks. This makes sense. $\endgroup$ – user1351484 Sep 25 at 4:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.