3
$\begingroup$

Can someone help review my analysis below and provide any feedback?

Problem

In a Computer Science class of 30 people comprising of 6 females, is there statistical significance of gender discrimination if only 3 students failed and all 3 were female?

Approach

Since the sample size is relatively small, we can use Fisher's Exact test for a conservative answer. Null hypothesis (H0): Failing is independent of gender Alternative hypothesis (Ha): Failing is dependent on gender

2x2 contingency matrix

Gender Failed Passed Total
Male 0 24 24
Female 3 3 6
Total 3 27 30

Calculate the Fisher's exact test p-value for this scenario under H0

p = (24!*6!*3!*27!)/(0!*24!*3!*3!*30!) = 0.005

Conclusion

Since the p-value of 0.005 is an order of magnitude less than the significance level 0.05, we reject the null hypothesis and conclude there is a statistically significant association between gender discrimination and failing at the 0.05 significance level. Since Fisher's exact test is relatively conservative, other tests would result in the same conclusion.

Improve this question
$\endgroup$
6
  • 7
    $\begingroup$ When you say "gender discrimination" it's unclear if you mean that in the sense of unfair bias against a particular gender, or merely in the sense of there being a difference between men and women (the term usually alludes to the former). No statistical test will tell you the reason for association - the statistical test can tell you if gender is a discriminating factor for likelihood of passing, but it can't tell you if the instructor discriminates against women (or if the difference is explained by differing levels of prior CS experience between genders, for example). $\endgroup$
    – Nuclear Hoagie
    Sep 26 at 17:41
  • 4
    $\begingroup$ @crosslingual The calculations are correct (doing it in R yields a similar p value) but as nuclear hoagie mentioned, the correctness of the computations is not evidence of discrimination. $\endgroup$
    – Demetri Pananos
    Sep 26 at 18:00
  • 3
    $\begingroup$ Just adding my voice. Showing discrimination is done in court. It uses statistical evidence. But that evidence a) Isn't proof in the mathematical sense, although it may be part of proof in either of the two legal senses and b) Is a whole lot more complicated than a single chi square test. You would need to also show evidence that the women were as prepared and did as much work as the men. It gets very complicated. $\endgroup$
    – Peter Flom
    Sep 26 at 19:21
  • 3
    $\begingroup$ One additional point. Your calculations are correct as long as the assumptions behind the test are correct. One of those assumptions is independence. Were the students and their grades independent of each other? If the students were put into (or self selected into) study groups or graded on small group projects, then they would not be independent. If the 3 students that failed were in a study group together, then something that happened (or did not happen) within the group would influence the whole group and more complicated calculations would be needed to be correct. $\endgroup$
    – Greg Snow
    Sep 26 at 20:27
  • 1
    $\begingroup$ The evidence for discrimination such as it were is that there are 6 female students in a class of 30. (Or perhaps that's evidence for implicit bias in whoever came up with the question.) $\endgroup$
    – dipetkov
    Sep 27 at 13:11

2 Answers 2

Reset to default
3
$\begingroup$

As well as the issues already raised in the comments and the answer - if you were motivated to conduct this test just because you noticed the gender difference then you are HARKing (hypothesizing after the results are known). This would compromise the validity of your test.

Unlikely things happen quite a lot by chance. If you didn't specify before seeing the results that you were planning to test this class and only this class for failure rate by sex then you are at risk of interpreting a coincidence as being something sinister.

For a more concrete example, suppose I have 100 coins and toss them each six times. If I notice that one coin gave me six heads I might test for bias in the data from this coin, and I would get a p-value of 0.03. Could I claim that this coin is biased? Obviously not, because I know that if I start with 100 unbiased coins I am likely to get 6 heads at least once, just by chance. I only did one statistical test, but it was motivated by this unusual observation, and interpreting this p-value in isolation could have led to the wrong conclusion.

To interpret your p-value the hypothesis must have been formed before the data was generated or collected.

Improve this answer
$\endgroup$
1
3
$\begingroup$

Your calculations are correct, as long as assumptions are correct (which may be not the case, as noted by Greg Snow in the comment section). However, one important problem is that this is quite a small sample size, so drawing conclusions about your population of interest may be trickier than you think. In particular, due to this small sample size, you may be largely overestimating the difference between men and women, and you might even be drawing a completely erroneous conclusion about which group is more likely to fail the exam.

When reading your conclusion, one wonders what course of action should be taken next. You don't mention any previous study about the issue at hand, or what is the motivation for conducting this study in the first place. If this a real-life problem and not an exercise from a textbook, you should be able to express this motivation in everyday language, without using statistical terms like "statistical significance" and whatnot.

By the way, it is a bit unclear what is the population of interest you're trying to draw conclusions on. When you write a general statement such as "there is a statistically significant association between gender discrimination and failing ", what is the population you are referring to? You should clarify that.

It is also unclear what you mean by discrimination, as pointed out in the comment section. If you mean "Are women discriminated against by this instructor?", the Fisher's exact test and the study design would not allow to say that, as you need to control for many other factors - in particular when discrimination can take root in long-past events and take many different forms, possibly beyond the university or instructor's control. When reading your conclusion, someone hasty and unfamiliar with statistics might be tempted to say that the instructor is discriminating against women, while the data and the test are largely insufficient to make such a statement. You should clarify that too.

But leaving all that aside for a moment, are you really interested in testing the null hypothesis of no difference between men and women? If before looking at the table you already knew that the null hypothesis was not true, then you're not really interested in rejecting the null hypothesis, but you are interested in assessing the difference between men and women.

The difference of percentage we observe between men and women in your sample is pretty large (0% failing among men vs. 50% failing among women). Do you think it is plausible that it reflects well what would happen if we multiplied the sample size by 100?

30 observations may make the test underpowered to detect the "real" underlying difference between men and women. When you get a small p-value with an underpowered study, the difference we see in the sample will be exaggerated and may be even in the wrong direction.

With a sample size of 6 women and 24 men and a criteria p < 0.05, the smallest absolute risk difference (percentage of women failing minus percentage of men failing) that would lead to a significant result is 33%: 2 women failing out of 6, and 0 men failing out of 24 (or alternatively, 24 men failing and 4 women failing).

Below is a histogram showing all possible observable risk differences with a sample size like yours, with a p-value < 0.05. You see that with this kind of sample, a risk difference between -33% and +33% never has a p-value < 0.05.

Histogram showing all the possible observed risk differences with a sample, for p < 0.05

So if the difference in failing between men and women exists, but is in reality anywhere between -33% and +33%, this sample size would not allow to assess its real magnitude (and possibly its direction; if the magnitude of the real risk difference is small enough, your sample with a significant p-value may be showing that women are more likely to fail than men when in reality they might be less likely to fail -see type S error).

Your test and conclusion probably calls for collecting more data and looking at previous studies, to give your study more context and avoid too hasty conclusions relative to what is happening in this class (besides the obvious underrepresentation of women).

Does the risk difference observed in your sample (50%) plausibly reflect the "real" underlying risk difference between men and women, regarding failing the exam? You should report the risk difference observed in your sample, and mention what previous studies say on the subject.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ A test being underpowered means that failing to reject the null hypothesis is basically meaningless, as you didn't collect enough data to have meaningful statistics in the first place. But when you get a small p-value in an underpowered study, that suggests the effect is so overwhelming that even a paucity of data can reveal it. I don't really follow your example with the histogram, it is no surprise that we make Type I errors by rejecting the null when it it true, that happens no matter what your sample size is. $\endgroup$
    – Nuclear Hoagie
    Sep 27 at 14:19
  • $\begingroup$ @NuclearHoagie My example assumes the null is false (which is often the case in social sciences). So ultimately we are not really interested in type 1/type 2 errors, but we are interested in assessing the magnitude (and direction) of the difference between men and women (is there a difference of 5%? 10%? 50%?). If we are interested in detecting if women are less likely than men to fail the exam (in the example, 5% vs. 10%), with this kind of sample (24 men, 6 women), tables with small p-values will always show that women are more likely to fail even if it's not the case in reality. $\endgroup$
    – J-J-J
    Sep 27 at 14:26
  • $\begingroup$ @NuclearHoagie My point is that we don't have enough data here to say anything else than "we can reject the null hypothesis of no difference between men and women -but we can't say much about the magnitude or the direction of this difference" . It is probably of limited interest. $\endgroup$
    – J-J-J
    Sep 27 at 14:34
  • $\begingroup$ @NuclearHoagie I rewrote the last part of the answer, hopefully this should clarify the point I was making. I don't think we disagree in principle. $\endgroup$
    – J-J-J
    Sep 28 at 8:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.