Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
We did a small project at school and I would like to analyse the data. We conducted a survey of simple yes/no questions and I would like to see if there is a difference between the responses from males and females.
For example, if I have 12 'yes' and 8 'no' answers from males and 4 'yes' 16 'no' answers from females, what test or procedure could I use to show that the difference is statistically significant?
For such small sample sizes I usually resort to Fisher's exact test. The wikipedia page actually has a worked out example that is very much like yours.
Asking if there is a difference is kind of a vague question, because there most likely will be a difference, even if it might be small and there will always be the ambiguity to distinguish 'real' differences (in your example between males and females) from bad luck in that particular sample.
Therefore i think it's better to either quantify the question (e.g. what is the chance to determine the gender from a yes/no answer?) or by just visualizing the data, which shows the earlier mentioned ambiguity between the sample uncertainty and the 'real' possibly underlying differences.
I'm going for the second option here and show one way of visualizing the observations.
The real data you have here is the yes/no answers to your survey for the two groups. Since we have only two options, we can choose a binomial model for each gender, which assumes there is an underlying probability of answering the survey question with yes, one for each gender (let's call them $p_m$ and $p_f$).
Then in a bayesian framework, this leads us to a Beta distribution as the posterior distribution for the binomial 'yes'-probability (see e.g. here under binomial distribution in the table). This basically tells us, that, given the observed yes/no counts, how much we should then believe, that the yes-probability was $p$. There is the additional choice of a prior, which can make it seem more complicated, so for the sake of simplicity i just chose the Jeffrey's prior, which essentially assumes no prior knowledge/bias about the yes-probabilities, so you could also say, it's fair or objective.
We can do this for both observed yes/no count pairs and plot both posteriors in one plot and get:
Now we can see, that there is a distinctive separation between the means of both posteriors, which means that there is good reason to believe that there is an actual difference between males and females here. But we can also see that there is a not so insignificant overlap around $p\approx0.4$, which is a visualization of the possible scenario that there might be no big difference between the two groups (both coming from $p\approx0.4$)), and we just had some bad luck in our sample!
We now could go on and answer quantified questions, like the one stated earlier or e.g. how high the chance is that the probabilities between males and females differ by more than 10% ($|p_f-p_m|>\frac{1}{10}$) by integrating the area under the joint posterior where the condition is true.
The joint posterior looks like this:
Here the purple area corresponds to scenarios where male and female probabilities differ more than 10% (and the blue area where they are closer to each other than 10%). By integrating the posterior under the purple area we find out that we should rationally believe that the scenario that males and females underlying yes-probability differs by more than 10% has 97.5% credibility, while the scenario that they are close to each other is only 2.5% credible. We now have a quantified answer to our specific quantified question, which makes it much clearer what we found out and doesn't leave much room for ambiguity (compared to the original question if they differ at all).
If we're not satisfied with the certainty we get out of this, we could also get more evidence by extending our survey and taking more samples to get a clearer picture (either sharper peaks with the same separation or converging peaks).
I don't see any great issue with using the ordinary chi-squared test (probably with Yates' correction) or the G-test for this circumstance.
The question of whether your sample would constitute a random sample rather than say a convenience sample may be more of an issue; that would be an issue for any test (and is a common problem for studies of this kind, for all that they're widely used).
Statistics can suggest many poor-quality conclusions, if the tests aren't done critically (in the sense of critical thinking).
There is now a lot of research suggesting that many differences which were thought to suggest differences between genders, actually show poor experimental design, failure to consider other explanations and confounding factors, social pressures and cognitive biases not eliminated in the experiment or its evaluation, etc.
You might want to learn from these and be very careful in your experimental design, otherwise whatever statistics and analysis you do, won't be worth anything.
