A typical example how bias in data is being copied by AI is Amazon's recruiting tool that got abandoned in 2018.

In the various reports it is implicitly (or sometimes explicitly) stated that the AI magnified the bias that was present in the data.

  • For instance, it is mentioned that there was a lot imbalance in the data, among the present employees there are many more men than women (and somehow the AI should translate this into women having lower probability to be successful candidates).

    That is because Amazon’s computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.

  • Also, the AI was focusing on patterns that negatively biased women, e.g. "women’s chess club captain" doing worse than "chess club captain".

    In effect, Amazon’s system taught itself that male candidates were preferable. It penalized resumes that included the word “women’s,” as in “women’s chess club captain.” And it downgraded graduates of two all-women’s colleges, according to people familiar with the matter. They did not specify the names of the schools.

To me this seems difficult to grasp intuitively. If the neural network AI is doing different (by amplifying) than the human recruiters or whatever standard has been used to train and validate, wouldn't that decrease the score on the cost function? If the AI tendency to recruit very few women (some statements say almost none) is considered bad, then why does the AI do this? Somehow this must be baked into the cost function or goal/object (and it is not the fault of AI, but of humans making bad cost functions).

I do see how some small imbalance in the data could be amplified by a dichotomous classification. Say that the conditional probability to get hired is (being based on biased data which already favors men) slightly in favor of men (ie. real human recruiters are already slightly more likely to hire men).

gender   property     estimated probability to be hired
         A  B    
m        1  1         90%
w        1  1         85%
m        1  0         55%
w        1  0         50%
m        0  1         25%
w        0  1         20%
m        0  0         6%
m        0  0         1%

Then a dichotomous classifier (optimizing a cost function like number of successful predictions) might exaggerate these differences and draw a hard border by saying that only everybody who scores above x% (depending on the cost function and relative weights of false negative and false positive) is classified, and this could lead to say only those who are male and have properties A and B are being classified as potential candidate.

However with more and more additional properties, besides woman/man, this effect would become more and more diffuse and less important. (Unless the boundary is placed at very high probabilities to be successful candidate and the comparison is made in the tails)

Was AI really so much biased (in relation to human recruiters), or was that aspect of amplification just a blown up media story?

If AI was amplifying the bias, then: How did it do so? Why could it not use the loads of information to make good predictions for women as well? Was the target that was being optimized (e.g. number of correct predictions) not correctly defined, or based on very extreme high probability for success and this hard classification cut-off boundary?


This is not a full answer. In the question I mentioned one way how I can imagine a small bias being amplified. In this question I write it down in more detail. I consider this way of amplification a bit trivial and wonder whether there are more reasons why Amazon's recruitment tool was considered biased, especially why it was biased because AI was being used and amplified the bias in the data.

Say that some model is fitted to predict a probability that a 'candidate in a particular class' is being hired/successful or not (like logistic regression). In these probabilities there might be 'only' a slight bias against women due to being based on biased history of hiring.

We could plot this as a distribution by ranking the classes that are occurring in the population according to how high the probability of success is and as a function of this probability describe how many people are in a related class.

Below this is done for men and women separately with hypothetical beta distributions (parameters 3,3 and 2.75,3.25) and for these women have a slight disadvantage and, on average, have a ~8% less probability to be hired.

Sidenote: these distributions might in practice/reality be more like discrete distributions

small difference - big difference

The big problem with these numbers occurs when some classification is being made in the tails.

E.g. if this model would be used in proactive recruitment and is used to scan profiles on large databases of potential recruits. For such purpose many candidates are being scanned and we do not want a large false positive error rate, so say that we set a boundary at 0.9 probability that a recruit would be successful. In that case we see that suddenly there are double as many male candidates as female candidates.

Is there something else?

For me the problems in this way of recruitment are not in the inner mechanics of some algorithm and some black box algorithm that we do not understand that is making magical mistakes and wrongly selecting candidates because it is misinterpreting words like "woman's chess club" (the estimates of probability might be very accurate, so no mistake there) or can't handle unbalanced data.

But instead it is due to the dichotomous classification which only selected people from the tails. That is a human choice and not intrinsic to the fact of using an algorithm or human. When Amazon is replacing the recruitment tool by humans then it might just again become biased (or even more biased). Among high potential recruits a human might just as well place mostly men (or at least that appears to be what they did in history).

It are the human recruiters that, for certain given properties (that relate to a high probability success), bias men in favor of women. This bias has not been amplified, it is only filtered out by the strict choice of only recruiting high potential candidates.

Are there other modes that amplify bias?

Edit, interaction with unbalanced data and the effect of selecting tails

After writing this answer, I realize now that there is an interaction in this 'comparison of tails' and 'unbalanced data'. In the example above I use two distributions with different means, but it would also occur if the distributions have different variance (or it will especially occur in those cases and the means don't need to differ at all).

This difference in variation could occur when some machine learning algorithm is trained on mostly male resume's which makes it very good at classifying men (e.g. relating mostly to activities that correlate with being men like certain sports activities or membership of gender specific clubs). Such training on men is creating a large variance in the predicted probabilities (the model is good at classifying men), and being able to assign to men more often extreme probability categories.

Then the scores that are generated by the model could look like:

example with different variance

On average, the model/algorithm does not assign a different probability of recruitment success to women, but it might place women more often in the middle categories which might be less interesting to recruiters.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.