What is the simplest statistical analysis to compare 2 groups on a simple survey?

Question

I designed a simple survey about cancer knowledge among adults. There will be two groups: Those that have first-hand experience with cancer (you or someone in your immediate family with cancer) and those without (negative on the above). There are seven questions related to cancer knowledge, each with one right answer. My question is: What is the simplest and most easily understood statistical analysis for this type of data?

Answer 1

Often with surveys there is a scoring function. You score the seven responses, sum them and compare the sums between the groups. The Wilcoxon rank sum test can then be used to look for differences between these sums (or averages). If the scores could be assumed to be approximately normally distributed a two sample student t test could be applied instead.

Answer 2

Is there more than 1 wrong answer for each question? Are you interested in different wrong answers? i.e. if the correct answer is A, but one group chose mostly B and the other group mostly chose C is that interesting? or is only the proportion correct of interest?

You could do a $\chi^2$ test or Fisher's exact test on the table of each group by answer (or correct status) for each question.

If you want to look at all the questions together then you can look at the larger contingency table (loglinear models, logistic, or poisson regression), or just compare an overall score between groups (t-test or non-parametric equivalent).

Answer 3

"The simplest and most easily understood statistical analysis" would not involve any hypothesis test at all, since these involve the use of p-values which are routinely misunderstood even by researchers. How about simply showing the number of correct scores for each person, distinguishing the two groups:

Answer 4

It depends on what are the 7 questions asking. If it's a validated questionnaire that you borrowed somewhere, and it specifically explores "knowledge," then I'll agree to take a sum or average and then report it by group.

If the questions are put together by you and they are quite different in the knowledge domain (for instance, a mixture of questions asking about prevention, detection, diagnosis, prognosis, treatment, and maintenance), then I would suggest starting with looking at the rate of being correct at each question individually by group. Perhaps you may find some insights (like both groups know equally well about prevention and detection, but the cancer patient group knows much more about diagnosis) along the way. This interesting information could be masked away quickly if the scores were summed in a rush.

Answer 5

Let's start with a prototype statement you'd like to make at the end, maybe "The first hand group answered an average of 62% of the questions right, well above the 42% for the inexperienced group".

Score each question as 1/0 so you can total 7/7 = 100%, 6/7 = 86%, and so forth. For a statistical test, compare the means of each group on these scores by a two-sample t test.

This approach seems simple and easily understood.

When you look at individual questions, you can use a proportions test to compare the proportions in the two groups who answer correctly.

Answer 6

The most accurate test is the t-test and it is also very simple. But the simplest one and surprisingly accurate, is the sign test.

Fuddy daddies may say that,strictly speaking,that test is not applicable in this situation, but I have used it in many similar situations and always got valid results.