Binomial distribution for gender discrimination?

Question

I could be on the wrong path, but I'm trying to use binomial distribution to identify gender discrimination and am having issues getting consistent results based on trial size(?)... I think. Small numbers work as expected, large numbers do not.

An example that works: 16 people are hired. 2 are women. What is the chance this happened randomly? (x)=2, (n)=16, (p)=.5. Example below [from R] show what I expected.

pbinom(2, size=16, prob=0.5) 
#0.0021 -> .2% chance that there is no gender discrimination

An example that doesn't work: 1150 people are hired. 350 are women. What is the chance this happened randomly? (x)=350, (n)=1150, (p)=.5. Example below [from R] does not show what I expected.

pbinom(350, size=1150, prob=0.5) 
# 2.388788e-41 -> ?!

Thanks in advance. Any suggestions better approached please!

Answer 1

Assume that one is hiring from a large pool of equally qualified applicants of whom half are women and half are men. The number of women hired out of $n$ is $X.$ Suppose that $p$ is the probability that any one hire will be a women.

Perhaps you want to test The null hypothesis $H_0: p = 1/2$ against $H_a: p < 1/2.$

For $n = 16, x = 2$ the P-value of this test is given in R as follows:

binom.test(2, 16, .5, alt="less")

        Exact binomial test

data:  2 and 16
number of successes = 2, number of trials = 16, p-value = 0.00209
alternative hypothesis: 
 true probability of success is less than 0.5
95 percent confidence interval:
 0.0000000 0.3438252
sample estimates:
probability of success 
                 0.125 

The P-value can also be found as follows:

pbinom(2, 16, .5)
[1] 0.002090454

Perhaps a more even-handed approach would be to test $H_0: p= 1/2$ against $H_a: p\ne 1/2,$ perhaps allowing for the possibility that the hiring process might be biased in either direction. It turns out that the P-value of this 2-sided test is twice the P-value of the 1-sided test.

binom.test(2, 16, .5)

        Exact binomial test

data:  2 and 16
number of successes = 2, number of trials = 16, p-value = 0.004181
alternative hypothesis: 
 true probability of success is not equal to 0.5
95 percent confidence interval:
 0.0155136 0.3834762
sample estimates:
probability of success 
                 0.125 

A major difficulty in interpreting results of these tests arises in justifying the assumption that "a large pool of equally qualified applicants of whom half are women and half are men" was used--or is available. Without addressing such much more difficult issues, counts of women hired may lead to 'statistical significance', but not likely to persuasion. @whuber's Comment provides useful advice.

For data with 350 women hired out of 1150, the P-values of one- and two-sided tests are shown below. Both of the P-values are very nearly $0,$ indicating that the observed $\hat p =X/n = 350/1150 = 0.304$ is inconsistent with $p = 1/2.$

binom.test(350, 1150, .5, alt="less")$p.val
[1] 2.388788e-41
binom.test(350, 1150, .5)$p.val
[1] 4.777577e-41

Answer 2

Bruce's answer is great. I'd like to provide another way of interrogating whether the results you've observed are reasonable. It's easy to look at a p-value and think it's "wrong" with respect to our intuitions about the observed data and our model.

It might help to reframe this by thinking about what data our model would generate under the null hypothesis. As whuber pointed out, gender bias in hiring is a complex topic, so I'm referring here to "number of heads", as in the number of coin flips that come up heads. However in principle the same issues would apply to any binomial model given appropriate assumptions are met.

First, let's simulate what number of heads we get if we flip 16 coins in a row, and repeat that simulation 10,000 times. What's the distribution of results, and where does 2 lie on that distribution?

a <- rbinom(10000, size = 16, prob = 0.5)
hist(a,
    breaks = "FD",
    xlab = "Number of heads",
    main = "Histogram of number of heads when n=16, p=0.5"
)
abline(v = 2, lty = "dashed")

2 is present in our simulated data but at a pretty low frequency. Therefore .2% seems at least in the right ballpark. Bear in mind, we're only doing 10,000 replicates, so there will of course be error.

Now, let's simulate what number of heads we get when simulating 1150 flips, repeat that process 10,000 times, and visualise the distribution along with your observed value of 350:

b <- rbinom(10000, size = 1150, prob = 0.5)
hist(b,
    breaks = "FD",
    xlab = "Number of heads",
    main = "Histogram of number of heads when n=1150, p=0.5"
)
abline(v = 350, lty = "dashed")

Huh. 350 isn't even visible on the distribution unless we manually adjust the x-axis!

## in fact 350 isn't visible unless we set xlim
hist(b, breaks = "FD",
    xlab = "Number of heads", xlim = c(340, max(b) * 1.1),
    main = "Histogram of number of heads when n=1150, p=0.5"
)
abline(v = 350, lty = "dashed")

This shows that for a binomial distribution with $p=0.5$ and $n=1150$, $x=350$ is a really weird result! Therefore an extremely small p-value isn't surprising. I think you would need in the range of 1e40 simulations to observe one value that extreme, in fact...

Answer 3

Before you get to the statistical mechanics this this type of test, you need to step back and make sure you remember the injunction that "correlation is not cause". Gender discrimination is one possible cause of non-equal hiring probabilities but there are many other possible causes, most commonly involving correlation between gender and relevant skills or qualifications for the position. Moreover, for the data in your question, we don't even have the number of applicants of each gender, so the probabilities we can estimate are not even conditional on applying for the position --- i.e., they are not "hiring probabilities" at all; they are joint probabilities of applying and then being hired. If males and females have unequal probabilities, that does not prove gender discrimination.

For this reason, it is best to do this type of analysis by first framing it in purely statistical terms where you are testing whether or not the probability of applying-and-being-hired is the same for a male or a female. For the data you have this would typically be done using a two-sided binomial test.$^\dagger$ Using the smaller dataset you can see from the test below that there is evidence of a non-equal joint probability for this event. (For the larger dataset the evidence for an unequal probability is much stronger).

#Perform test for equal probability
binom.test(2, 16, conf.level = 0.99, alt = "two.sided")

            Exact binomial test

data:  2 and 16
number of successes = 2, number of trials = 16, p-value = 0.004181
alternative hypothesis: true probability of success is not equal to 0.5
99 percent confidence interval:
 0.006658398 0.462758698
sample estimates:
probability of success 
                 0.125 

Now that you have established evidence of non-equal probabilies you can have a think about what, if any, causal inferences you can make. In particular, you will need to think about whether there is anything in the experimental setup that would allow you to reject other causal explanations for unequal probabilities (e.g., men more likely to apply, men more likely to have required skills for the position, etc.). Without more, it would be extremely dubious to use this statistical evidence to conclude that gender discrimination has occurred.

If you would like to read more about this topic, I recommend reading some studies on gender discrimination in hiring in labour economics. Studies on this topic typically either conduct randomised controlled trials (RCTs) using fake CVs with randomised sex assignment, or they use observational data and attempt to filter out confounding variables using regression methods. Another method is to look at skill and qualification metrics of people within the organisation, filtering for their organisational level and other covariates, and then see if there is any residual correlation between the skills/qualifications and gender; if there is, this can suggest gender discrimination against the group with higher residual skill values (i.e., these are higher because the discrimination causes a more strict filter for hiring/promotion for people in that group). This is quite a complicated field, and it involves much more than the kinds of gross statistical comparisons in output that are being compared here. It generally involves deeper thinking about relationships between cause and statistical association.

---

$^\dagger$ Some analysts will recommend a one-sided test, since you with to test the alternative that women have a lower probability of applying-and-being-hired. My view is that two-sided tests should be performed, so as not to bias the testing in favour of a hypothesis that might have been influenced by the data.

Answer 4

The results are consistent.

The differences in p-values that you get with trial size is as expected.

The dependency of the result on the parameter $n$ occurs because the distribution becomes more narrow.

The same relative deviation becomes less probable when $n$ increases.

See the below example (adapted from here) for the distribution of the observed number of women $k$ and the fraction of women $f$ as a function of the number of workers $n$.

Related question is How to estimate a probability of an event to occur based on its count?

---

Consistency

The fact that the same effect size (e.g ratio of women) becomes more significant is a good thing.

This relates to 'statistical consistency'*, and this is a desirable property. It means that we can make estimates better (as much as we want) by collecting larger samples.

So that is what you did with your sample of 1150, it is a more accurate estimate of the ratio of women than the sample of 16.

---

*An estimator is consistent if 'The probability for the estimator to differ by a certain quantity from the true value approaches zero, when we increase the sample size'.

---

Sidenote: Strictly speaking you only observed that the ratio of women is different from 0.5. But is this also discrimination?

---

Sidenote: This comparison of the 'deviation from the middle' is at the origin of p-values and the normal distribution. In 1710 similar comparisons were made by Arbuthnot comparing ratios in the birth of boys and girls. It is one of the first cases of expressing a p-value.

Answer 5

In basic hypothesis testing, we articulate a null hypothesis, choose an ordering of results, collect data, and then find the probability, given the null hypothesis, of getting results that are, according to the ordering we chose, as extreme or more extreme as the results that we found.

That's it. We find the probability of the results given the hypothesis. Nothing else.

We do not find the probability of the hypothesis given results. We do not find the probability of some hypothesis other than the null hypothesis.

In your case, the null hypothesis is that each hiring decision is a Bernoulli trial, each one is independent of the others, and the probability for each is 0.5. Your ordering is "more men is more extreme". The number 0.0021 you found is the probability of getting as many men or more, given those premises. It is not the probability of there being no gender discrimination. Not only are you confusing $P(E|H)$ and $P(H|E)$, there are many alternative hypotheses other than "there in gender discrimination". For instance, there could be fewer qualified female candidates, or the hirings could be not independent.

As for the effect of larger sample size, the number of standard deviations scales with the square root of the sample size. That is, in the examples you gave, the sample size was multiplied by 71.875, and square root of that is 8.48, so deviations from the expected mean of 0.50 represent 8.48 as many standard deviations.