Was Amazon's AI tool, more than human recruiters, biased against women?

Question

A typical example how bias in data is being copied by AI is Amazon's recruiting tool that got abandoned in 2018.

In the various reports it is implicitly (or sometimes explicitly) stated that the AI magnified the bias that was present in the data.

• For instance, it is mentioned that there was a lot imbalance in the data, among the present employees there are many more men than women (and somehow the AI should translate this into women having lower probability to be successful candidates).

  That is because Amazon’s computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.

• Also, the AI was focusing on patterns that negatively biased women, e.g. "women’s chess club captain" doing worse than "chess club captain".

  In effect, Amazon’s system taught itself that male candidates were preferable. It penalized resumes that included the word “women’s,” as in “women’s chess club captain.” And it downgraded graduates of two all-women’s colleges, according to people familiar with the matter. They did not specify the names of the schools.

To me this seems difficult to grasp intuitively. If the neural network AI is doing different (by amplifying) than the human recruiters or whatever standard has been used to train and validate, wouldn't that decrease the score on the cost function? If the AI tendency to recruit very few women (some statements say almost none) is considered bad, then why does the AI do this? Somehow this must be baked into the cost function or goal/object (and it is not the fault of AI, but of humans making bad cost functions).

---

I do see how some small imbalance in the data could be amplified by a dichotomous classification. Say that the conditional probability to get hired is (being based on biased data which already favors men) slightly in favor of men (ie. real human recruiters are already slightly more likely to hire men).

gender   property     estimated probability to be hired
         A  B    
m        1  1         90%
w        1  1         85%
m        1  0         55%
w        1  0         50%
m        0  1         25%
w        0  1         20%
m        0  0         6%
m        0  0         1%

Then a dichotomous classifier (optimizing a cost function like number of successful predictions) might exaggerate these differences and draw a hard border by saying that only everybody who scores above x% (depending on the cost function and relative weights of false negative and false positive) is classified, and this could lead to say only those who are male and have properties A and B are being classified as potential candidate.

However with more and more additional properties, besides woman/man, this effect would become more and more diffuse and less important. (Unless the boundary is placed at very high probabilities to be successful candidate and the comparison is made in the tails)

Was AI really so much biased (in relation to human recruiters), or was that aspect of amplification just a blown up media story?

If AI was amplifying the bias, then: How did it do so? Why could it not use the loads of information to make good predictions for women as well? Was the target that was being optimized (e.g. number of correct predictions) not correctly defined, or based on very extreme high probability for success and this hard classification cut-off boundary?

Answer 1

This is not a full answer. In the question I mentioned one way how I can imagine a small bias being amplified. In this question I write it down in more detail. I consider this way of amplification a bit trivial and wonder whether there are more reasons why Amazon's recruitment tool was considered biased, especially why it was biased because AI was being used and amplified the bias in the data.

Say that some model is fitted to predict a probability that a 'candidate in a particular class' is being hired/successful or not (like logistic regression). In these probabilities there might be 'only' a slight bias against women due to being based on biased history of hiring.

We could plot this as a distribution by ranking the classes that are occurring in the population according to how high the probability of success is and as a function of this probability describe how many people are in a related class.

Below this is done for men and women separately with hypothetical beta distributions (parameters 3,3 and 2.75,3.25) and for these women have a slight disadvantage and, on average, have a ~8% less probability to be hired.

Sidenote: these distributions might in practice/reality be more like discrete distributions

The big problem with these numbers occurs when some classification is being made in the tails.

E.g. if this model would be used in proactive recruitment and is used to scan profiles on large databases of potential recruits. For such purpose many candidates are being scanned and we do not want a large false positive error rate, so say that we set a boundary at 0.9 probability that a recruit would be successful. In that case we see that suddenly there are double as many male candidates as female candidates.

---

Is there something else?

For me the problems in this way of recruitment are not in the inner mechanics of some algorithm and some black box algorithm that we do not understand that is making magical mistakes and wrongly selecting candidates because it is misinterpreting words like "woman's chess club" (the estimates of probability might be very accurate, so no mistake there) or can't handle unbalanced data.

But instead it is due to the dichotomous classification which only selected people from the tails. That is a human choice and not intrinsic to the fact of using an algorithm or human. When Amazon is replacing the recruitment tool by humans then it might just again become biased (or even more biased). Among high potential recruits a human might just as well place mostly men (or at least that appears to be what they did in history).

It are the human recruiters that, for certain given properties (that relate to a high probability success), bias men in favor of women. This bias has not been amplified, it is only filtered out by the strict choice of only recruiting high potential candidates.

Are there other modes that amplify bias?

---

Edit, interaction with unbalanced data and the effect of selecting tails

After writing this answer, I realize now that there is an interaction in this 'comparison of tails' and 'unbalanced data'. In the example above I use two distributions with different means, but it would also occur if the distributions have different variance (or it will especially occur in those cases and the means don't need to differ at all).

This difference in variation could occur when some machine learning algorithm is trained on mostly male resume's which makes it very good at classifying men (e.g. relating mostly to activities that correlate with being men like certain sports activities or membership of gender specific clubs). Such training on men is creating a large variance in the predicted probabilities (the model is good at classifying men), and being able to assign to men more often extreme probability categories.

Then the scores that are generated by the model could look like:

On average, the model/algorithm does not assign a different probability of recruitment success to women, but it might place women more often in the middle categories which might be less interesting to recruiters.

Answer 2

One additional thought, not fully explored: Maybe we should think whether overfitting can amplify existing bias.

In a linear model (where we understand better what happens), we observe overfitting with coefficents being of too large absolute size.

Increased bias against women then may be expressed as the coefficient for gender: female being too large (negative), not only compared to that we believe it should be zero, but also compared to the input data.
Or a set of coefficients for input variates that are correlated with gender have large size in opposite directions and do not perfectly cancel themselves, leading to a net outcome that gives a too-sharp distinction between female and male candidates.

Or, to put in in a different way: overfitted models tend to be overconfident in their predictions (not well-calibrated in a probabilistic sense).
If the "attitude" behind the training data is somewhat reluctant (e.g. due to sexistic bias) to hire women, the overconfident prediction may amplify this into confidently rejecting (most) women.

---

• In general, I believe we (machine learning community) don't do overly well in terms of preventing overfitting.

• I did not check whether this would lead to a bias against the minority class as in systematic effect over a large number of models, but:

• we have an additional selection effect/publication bias here: even if this effect only leads to high variance:

  • we have a number of minorities or protected classes* (sociologial meaning here). We are (rightly!) concerned if an existing bias against any of these is amplified by the AI.
  • But even if there were only the question of bias against women being amplified, and models were equally likely to amplify or deamplify (bias against men) the input data bias, given that there is historically a bias against women, those models would likely be judged differently. Amplified bias against women is bad (again: rightly so); Deamplified bias against women (or maybe even bias against men) is likely seen more leniently since it counteracts the existing bias.
    (Personally, I'm not so sure I'd agree with that lenience, I very much prefer countermeasures with negative feedback loop which stop exerting force when there is no more bias.)

  In any case, a model amplifying bias against women is IMHO more likely to be brought to our attention.

---

*I think it somehow weird if we women are considered a 50 % "minority".

Answer 3

Sample selection bias is introduced by the selection of individuals, groups, or data for analysis in such a way that the samples are not representative of the population intended to be analyzed.9 In particular, sample selection bias occurs during data analysis as a result of conditioning on some variables in the dataset (for example, a particular skin color, gender, among others), which in turn can create spurious correlations. For example, in analyzing the effect of motherhood on wages, if the study is restricted to women who are already employed, then the measured effect will be biased as a result of conditioning on employed women.9 Common types of sample selection bias include Berkson's paradox20 and sample truncation.9

Answer 4

• Also, the AI was focusing on patterns that negatively biased women, e.g. "women’s chess club captain" doing worse than "chess club captain".

  In effect, Amazon’s system taught itself that male candidates were preferable. It penalized resumes that included the word “women’s,” as in “women’s chess club captain.” And it downgraded graduates of two all-women’s colleges, according to people familiar with the matter. They did not specify the names of the schools.

This could be a problem where an AI is bad at solving problems. If a "chess club captain" is a bonus on a curriculum vitae then so will be a "women's chess club captain". A human recruiter will understand this immediately. A computer/AI recruiter will not understand this because it has much less understanding of complex patterns that it did not encounter before it has not been trained to 'understand'.

A more clear example is Roger Penrose's chess problem

For a human this is an obvious draw as black's pieces are all blocked and the three bishops are on black fields making them unable to capture white pieces on white fields.

For a computer this is apparently a very difficult problem. It does not understand chess in the way that we do. And it can not figure out the situation in this simple problem.

This can make computer/AI recruiters more harsh and stick to old patterns. They are less able to adapt to a situation where more women are applying for specific jobs whereas human recruiters can (if they want).

The situation of more and better women applying for jobs is a new situation like Roger Penrose's chess puzzle.