We did a small project at school and I would like to analyse the data. We conducted a survey of simple yes/no questions and I would like to see if there is a difference between the responses from males and females.

For example, if I have 12 'yes' and 8 'no' answers from males and 4 'yes' 16 'no' answers from females, what test or procedure could I use to show that the difference is statistically significant?

  • $\begingroup$ I like to use this hyperlink and look at overlap of 95% confidence intervals. A Wilson or Jeffrey's prior is more applicable in your sample sizes. $\endgroup$ – EngrStudent Sep 14 '17 at 15:24

For such small sample sizes I usually resort to Fisher's exact test. The wikipedia page actually has a worked out example that is very much like yours.

| cite | improve this answer | |
  • 4
    $\begingroup$ Just be aware of the multiple comparisons problem, and be aware that data dredging can mean that the numbers (or conclusions of "significant") you get from Fisher's exact test aren't necessarily as informative as you might like. (See this xckd cartoon for a humorous illustration of the problem.) $\endgroup$ – R.M. Sep 14 '17 at 15:45
  • $\begingroup$ Ah I missed that the OP was asking about "questions" rather than a single question and could thus could thus run into the multiple comparison problem. I wonder if any of the suggestions below allow to test multiple statements at the same time? (Much like anova vs pairwise t-tests) $\endgroup$ – Maarten Punt Sep 14 '17 at 17:42

Asking if there is a difference is kind of a vague question, because there most likely will be a difference, even if it might be small and there will always be the ambiguity to distinguish 'real' differences (in your example between males and females) from bad luck in that particular sample.

Therefore i think it's better to either quantify the question (e.g. what is the chance to determine the gender from a yes/no answer?) or by just visualizing the data, which shows the earlier mentioned ambiguity between the sample uncertainty and the 'real' possibly underlying differences.

I'm going for the second option here and show one way of visualizing the observations.

The real data you have here is the yes/no answers to your survey for the two groups. Since we have only two options, we can choose a binomial model for each gender, which assumes there is an underlying probability of answering the survey question with yes, one for each gender (let's call them $p_m$ and $p_f$).

Then in a bayesian framework, this leads us to a Beta distribution as the posterior distribution for the binomial 'yes'-probability (see e.g. here under binomial distribution in the table). This basically tells us, that, given the observed yes/no counts, how much we should then believe, that the yes-probability was $p$. There is the additional choice of a prior, which can make it seem more complicated, so for the sake of simplicity i just chose the Jeffrey's prior, which essentially assumes no prior knowledge/bias about the yes-probabilities, so you could also say, it's fair or objective.

We can do this for both observed yes/no count pairs and plot both posteriors in one plot and get:

survey posteriors

Now we can see, that there is a distinctive separation between the means of both posteriors, which means that there is good reason to believe that there is an actual difference between males and females here. But we can also see that there is a not so insignificant overlap around $p\approx0.4$, which is a visualization of the possible scenario that there might be no big difference between the two groups (both coming from $p\approx0.4$)), and we just had some bad luck in our sample!

We now could go on and answer quantified questions, like the one stated earlier or e.g. how high the chance is that the probabilities between males and females differ by more than 10% ($|p_f-p_m|>\frac{1}{10}$) by integrating the area under the joint posterior where the condition is true.

The joint posterior looks like this:

joint posterior

Here the purple area corresponds to scenarios where male and female probabilities differ more than 10% (and the blue area where they are closer to each other than 10%). By integrating the posterior under the purple area we find out that we should rationally believe that the scenario that males and females underlying yes-probability differs by more than 10% has 97.5% credibility, while the scenario that they are close to each other is only 2.5% credible. We now have a quantified answer to our specific quantified question, which makes it much clearer what we found out and doesn't leave much room for ambiguity (compared to the original question if they differ at all).

If we're not satisfied with the certainty we get out of this, we could also get more evidence by extending our survey and taking more samples to get a clearer picture (either sharper peaks with the same separation or converging peaks).

| cite | improve this answer | |
  • 1
    $\begingroup$ Thank you so much for the answer, that is way more than I had expected! $\endgroup$ – Amiris Sep 15 '17 at 8:17
  • $\begingroup$ @King You're welcome! Glad if i could make things clearer for you :) $\endgroup$ – Thies Heidecke Sep 15 '17 at 9:53
  • $\begingroup$ @ThiesHeidecke I would have a second question from my friend who did a small experiment. It goes into the same direction, but probably needs a bit more time. It is on my profile and I thought maybe you could give it a look. $\endgroup$ – Amiris Sep 20 '17 at 8:03

I don't see any great issue with using the ordinary chi-squared test (probably with Yates' correction) or the G-test for this circumstance.

The question of whether your sample would constitute a random sample rather than say a convenience sample may be more of an issue; that would be an issue for any test (and is a common problem for studies of this kind, for all that they're widely used).

| cite | improve this answer | |

Statistics can suggest many poor-quality conclusions, if the tests aren't done critically (in the sense of critical thinking).

There is now a lot of research suggesting that many differences which were thought to suggest differences between genders, actually show poor experimental design, failure to consider other explanations and confounding factors, social pressures and cognitive biases not eliminated in the experiment or its evaluation, etc.

You might want to learn from these and be very careful in your experimental design, otherwise whatever statistics and analysis you do, won't be worth anything.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.