I have questions inspired from the Amazon recent recruitment scandal, where they were accused of discrimination against women in their recruitment process. More info here:
Amazon.com Inc's machine-learning specialists uncovered a big problem: their new recruiting engine did not like women.
The team had been building computer programs since 2014 to review job applicants' resumes with the aim of mechanizing the search for top talent...
... The company's experimental hiring tool used artificial intelligence to give job candidates scores ranging from one to five stars...
... But by 2015, the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way.
That is because Amazon's computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry. (For a graphic on gender breakdowns in tech, see: here) In effect, Amazon's system taught itself that male candidates were preferable. It penalized resumes that included the word "women's," as in "women's chess club captain." And it downgraded graduates of two all-women's colleges, according to people familiar with the matter. They did not specify the names of the schools.
Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said.
The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project...
... The company's experiment... offers a case study in the limitations of machine learning.
... computer scientists such as Nihar Shah, who teaches machine learning at Carnegie Mellon University, say there is still much work to do.
"How to ensure that the algorithm is fair, how to make sure the algorithm is really interpretable and explainable - that's still quite far off," he said.
[Amazon] set up a team in Amazon's Edinburgh engineering hub that grew to around a dozen people. Their goal was to develop AI that could rapidly crawl the web and spot candidates worth recruiting, the people familiar with the matter said.
The group created 500 computer models focused on specific job functions and locations. They taught each to recognize some 50,000 terms that showed up on past candidates' resumes. The algorithms learned to assign little significance to skills that were common across IT applicants, such as the ability to write various computer codes...
Instead, the technology favored candidates who described themselves using verbs more commonly found on male engineers’ resumes, such as "executed" and "captured," one person said.
Let's say I want to build a statistical model to predict some output from personal data, like a five star ranking to help recruiting new people. Let's say I also want to avoid gender discrimination, as an ethical constraint. Given two strictly equal profile apart from the gender, the output of the model should be the same.
Should I use the gender (or any data correlated to it) as an input and try to correct their effect, or avoid to use these data?
How do I check the absence of discrimination against gender?
How do I correct my model for data that are statistically discriminant but I don't want to be for ethical reasons?