A typical example how bias in data is being copied by AI is Amazon's recruiting tool that got abandoned in 2018.
In the various reports it is implicitly (or sometimes explicitly) stated that the AI magnified the bias that was present in the data.
For instance, it is mentioned that there was a lot imbalance in the data, among the present employees there are many more men than women (and somehow the AI should translate this into women having lower probability to be successful candidates).
That is because Amazon’s computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.
Also, the AI was focusing on patterns that negatively biased women, e.g. "women’s chess club captain" doing worse than "chess club captain".
In effect, Amazon’s system taught itself that male candidates were preferable. It penalized resumes that included the word “women’s,” as in “women’s chess club captain.” And it downgraded graduates of two all-women’s colleges, according to people familiar with the matter. They did not specify the names of the schools.
To me this seems difficult to grasp intuitively. If the neural network AI is doing different (by amplifying) than the human recruiters or whatever standard has been used to train and validate, wouldn't that decrease the score on the cost function? If the AI tendency to recruit very few women (some statements say almost none) is considered bad, then why does the AI do this? Somehow this must be baked into the cost function or goal/object (and it is not the fault of AI, but of humans making bad cost functions).
I do see how some small imbalance in the data could be amplified by a dichotomous classification. Say that the conditional probability to get hired is (being based on biased data which already favors men) slightly in favor of men (ie. real human recruiters are already slightly more likely to hire men).
gender property estimated probability to be hired A B m 1 1 90% w 1 1 85% m 1 0 55% w 1 0 50% m 0 1 25% w 0 1 20% m 0 0 6% m 0 0 1%
Then a dichotomous classifier (optimizing a cost function like number of successful predictions) might exaggerate these differences and draw a hard border by saying that only everybody who scores above x% (depending on the cost function and relative weights of false negative and false positive) is classified, and this could lead to say only those who are male and have properties A and B are being classified as potential candidate.
However with more and more additional properties, besides woman/man, this effect would become more and more diffuse and less important. (Unless the boundary is placed at very high probabilities to be successful candidate and the comparison is made in the tails)
Was AI really so much biased (in relation to human recruiters), or was that aspect of amplification just a blown up media story?
If AI was amplifying the bias, then: How did it do so? Why could it not use the loads of information to make good predictions for women as well? Was the target that was being optimized (e.g. number of correct predictions) not correctly defined, or based on very extreme high probability for success and this hard classification cut-off boundary?