Avoiding social discrimination in model building

Question

I have questions inspired from the Amazon recent recruitment scandal, where they were accused of discrimination against women in their recruitment process. More info here:

Amazon.com Inc's machine-learning specialists uncovered a big problem: their new recruiting engine did not like women.
The team had been building computer programs since 2014 to review job applicants' resumes with the aim of mechanizing the search for top talent...
... The company's experimental hiring tool used artificial intelligence to give job candidates scores ranging from one to five stars...
... But by 2015, the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way.
That is because Amazon's computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry. (For a graphic on gender breakdowns in tech, see: here) In effect, Amazon's system taught itself that male candidates were preferable. It penalized resumes that included the word "women's," as in "women's chess club captain." And it downgraded graduates of two all-women's colleges, according to people familiar with the matter. They did not specify the names of the schools.
Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said.
The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project...
... The company's experiment... offers a case study in the limitations of machine learning.
... computer scientists such as Nihar Shah, who teaches machine learning at Carnegie Mellon University, say there is still much work to do.
"How to ensure that the algorithm is fair, how to make sure the algorithm is really interpretable and explainable - that's still quite far off," he said.

MASCULINE LANGUAGE
[Amazon] set up a team in Amazon's Edinburgh engineering hub that grew to around a dozen people. Their goal was to develop AI that could rapidly crawl the web and spot candidates worth recruiting, the people familiar with the matter said.
The group created 500 computer models focused on specific job functions and locations. They taught each to recognize some 50,000 terms that showed up on past candidates' resumes. The algorithms learned to assign little significance to skills that were common across IT applicants, such as the ability to write various computer codes...
Instead, the technology favored candidates who described themselves using verbs more commonly found on male engineers’ resumes, such as "executed" and "captured," one person said.

Let's say I want to build a statistical model to predict some output from personal data, like a five star ranking to help recruiting new people. Let's say I also want to avoid gender discrimination, as an ethical constraint. Given two strictly equal profile apart from the gender, the output of the model should be the same.

1. Should I use the gender (or any data correlated to it) as an input and try to correct their effect, or avoid to use these data?

2. How do I check the absence of discrimination against gender?

3. How do I correct my model for data that are statistically discriminant but I don't want to be for ethical reasons?

Answer 1

This paper provides an excellent overview of how to navigate gender bias especially in language-based models: Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings - Bolukbasi et. al.. A nice blog summary can be found here:

https://developers.googleblog.com/2018/04/text-embedding-models-contain-bias.html

You'll find a larger compendium of resources here:

https://developers.google.com/machine-learning/fairness-overview/

You'll find a slew of techniques in the above links to mitigate gender bias. Generally speaking they fall into three classes:

1) Under/Over sampling your data. This is intended to oversample high-quality female resumes and under sample male resumes.

2) Subtracting out the "gender subspace." If your model is gender-biased, then you could demonstrate it to be so by using your resume embeddings to directly predict gender. After building such an auxiliary model (even just sampling common terms belonging to either gender, and then applying PCA), you can in effect subtract out this dimension from the model, normalizing the resume to be gender-neutral. This is the main technique used in Bolukbasi's paper.

3) Adversarial Learning. In this case you try to generate additional data by trying to generate more versions of high quality female resumes that are otherwise indistinguishable from real resumes.

Answer 2

This is not an answer to your question but just a few thoughts that are too long to fit in a comment.

I think one problem we have to consider when thinking about these issues is that every model discriminates, and they will do so on the basis of any association present in the data. That is arguably the whole purpose of a predictive model. For instance, men are genuinely more likely to commit crime than women, so almost any model that has access to this information will draw such an inference.

But that doesn't mean we should convict someone partially on the basis of gender, even though a man will generally appear more likely to have committed a crime (other things equal). Rather we should require direct evidence of a crime when making such decisions, and not information on mere association. As another example: do people who are more likely to get sick really deserve to pay higher insurance premiums?

So when it comes to discrimination, I would argue that the issue deals more with ethical application, rather than models themselves being unfair. If we are worried about perpetuating discrimination or other unfair outcomes when using a model in given situation, then perhaps we should not be using a model.

Answer 3

I used to work on a project to develop software management best practices. I observed roughly fifty software teams in the field. Our sample was around 77, but we ended up seeing around a hundred teams. In addition to collecting data on things such as certifications, degrees and so forth, we also collected a variety of psychological and demographic data.

Software development teams have some very significant self-selection effects in it that, while having nothing to do with gender, are strongly correlated with gender. Also, managers tend to replicate themselves. People hire people they are comfortable with, and they are most comfortable with themselves. There is also evidence that people are being rated in a cognitively biased way. Imagine that, as a manager, I highly value prompt arrival at the start of work. I will then rate on that. Another manager, who just cares that the work gets done, may rate on something entirely different as important.

You noted that men use language differently, but it is also true that people with different personalities use language in different ways. There may be ethnic language usage differences as well, see for example the current controversy at Harvard and Asian admissions.

Now you assume that the software firms discriminate against women, but there is another form of gender discrimination going on in the software development industry that you haven’t accounted for. When you control for objective things such as certifications, degrees, tenure and so forth, the average woman earns 40% more than the average man. There are three sources of employment discrimination in the world.

The first is that managers or owners do not wish to hire someone on the basis of some feature. The second is that coworkers do not wish to work with the people with that feature. The third is that customers do not want people who have a feature. It appears the wage discrimination is being triggered by customers because the work product is different, and from the customers’ perspectives, also better. This same feature causes male dental hygienists to take lower pay than women. It is also seen in a bias toward “born here” in world soccer wages.

The best control for this is to understand your data and the social forces involved. Any firm that uses its own data will tend to replicate itself. That may be a very good thing, but it could also make them blind to forces at work. The second control is to understand your objective function. Profits may be a good function, but it may be a bad function. There are values in play in the selection of an objective loss function. Then, finally, there is the issue of testing the data against demographics to determine if unfortunate discrimination is happening.

Finally, and this is a bigger problem in things like AI where you cannot get good interpretative statistics, you will want to control for Yule’s paradox. The classic historical example is the discovery that 44% of men were accepted to UC Berkley while only 35% of women were admitted in 1973. This was a huge difference and statistically significant. It was also misleading.

This was obviously scandalous, and so the university decided to look at which were the offending majors. Well, it turned out that when you controlled for major, there was a statistically significant bias in favor of admitting women. Of the eighty-five majors, six were biased toward women and four toward men, the remainder were not significant. The difference was that women were, disproportionately, applying for the most competitive majors and so few of either gender were getting in. Men were more likely to apply to less competitive majors.

Adding in Yule’s paradox creates an even deeper layer for discrimination. Imagine, instead of a gender test, there was a gender test by type of job. You could possibly pass a company-wide gender neutral test but fail at the task level. Imagine that only women were recruited for V&V and only men for systems administration. You would look gender neutral, and you wouldn’t be.

One potential solution to this is to run competitive AIs that use differing objective criteria of “goodness.” The goal is to widen the net, not narrow it. This can also help avoid another problem in the management literature. While 3% of males are sociopaths, that number climbs substantially as you go further and further up the corporate ladder. You don’t want to be filtering for sociopaths.

Finally, you may not want to consider using AI for certain types of positions. I am job hunting right now. I am also sure I am being filtered out, and I haven’t figured out how to get around it. I am sitting on a very disruptive new technology. The problem is that my work doesn’t match the magic words. Instead, I have the next set of magic words. Right now, I am worth a fortune to the right firm, but in one case where I applied, I received an automated decline in less than a minute. I have a friend who has served as the CIO of federal agencies. He applied for a job where the hiring manager was waiting to see his application come through so he could pretty much be offered the job. It never came through because the filters blocked it.

This sets up the second problem of AI. If I can work out from online resumes who Amazon is hiring, then I can magic word my resume. Indeed, I am working on my resume right now to get it to fit non-human filters. I can also tell from the e-mails from recruiters that some parts of my resume are being zoomed in on and other parts ignored. It is as if the recruiting and hiring process has been taken over by software like Prolog. Logical constraints met? Yes! This is the optimal candidate or set of candidates. Are they optimal?

There isn't a pre-built answer to your question, only problems to engineer around.

Answer 4

What the Amazon story shows is that it is very hard to avoid the bias. I doubt that Amazon hired dumb people for this problem, or that they were lacking skills, or that they didn't have enough data, or that they didn't have enough AWS credits to train a better model. The problem was that the complicated machine learning algorithms are very good at learning patterns in the data, gender bias is exactly that kind of pattern. There was bias in the data, as the recruiters (consciously or not), favored male candidates. I'm not saying in here that Amazon is a company that discriminates job candidates, I'm sure they have thousands of anti-discriminatory policies and also hire pretty good recruiters. The problem with this kind of bias and prejudice is that exists no matter how hard you try to fight it. There are tons of psychology experiments showing that people may declare not to be biased (e.g. racist), but still make biased actions, without even realizing it. But answering your question, to have algorithm that is not biased, you would need to start with data that is free of this kind of bias. Machine learning algorithms learn to recognize and repeat the patterns they see in the data, so if your data records biased decisions, the algorithm will likely learn and amplify those bias.

Second thing is managing the data. If you want to prohibit your algorithm from learning to make biased decisions, you should remove all the information that would help if to discriminate between groups of interest (gender in here). This does not mean removing only the information about gender, but also all the information that could lead to identifying gender, and this could be lots of things. There are obvious ones like name and photo, but also indirect ones, e.g. maternal leave in resume, but also education (what if someone went to girls-only school?), or even job history (say that recruiters in your company are not biased, but what if every other recruiter before was biased, so the work history reflects all those biased decisions?), etc. As you can see, identifying those issues may be pretty complicated (another reason why Amazon may have failed).

As about questions 2. and 3., there is no easy answers and I do not feel competent enough to try answering them in detail. There is tons of literature on both prejudice and bias in society, and about algorithmic bias. This is always complicated and there is, unfortunately, no simple recipes for this. Companies, like Google, hire experts whose role is identifying and preventing this kind of bias in algorithms.

Answer 5

In order to build a model of this kind, it is important to first understand some basic statistical aspects of discrimination and process-outcomes. This requires understanding of statistical processes that rate objects on the basis of characteristics. In particular, it requires understanding the relationship between use of a characteristic for decision-making purposes (i.e., discrimination) and assessment of process-outcomes with respect to said characteristic. We start by noting the following:

• Discrimination (in its proper sense) occurs when a variable is used in the decision process, not merely when the outcome is correlated with that variable. Formally, we discriminate with respect to a variable if the decision function in the process (i.e., the rating in this case) is a function of that variable.

• Disparities in outcome with respect to a particular variable often occur even when there is no discrimination on that variable. This occurs when other characteristics in the decision function are correlated with the excluded variable. In cases where the excluded variable is a demographic variable (e.g., gender, race, age, etc.) correlation with other characteristics is ubiquitous, so disparities in outcome across demographic groups are to be expected.

• It is possible to try to reduce disparities in outcomes across demographic groups through affirmative-action, which is a form of discrimination. If there are disparities in process-outcomes with respect to a variable, it is possible to narrow those disparities by using the variable as a decision-variable (i.e., by discriminating on that variable) in a way that favours groups that are "underrepresented" (i.e., groups with lower proportions of positive outcomes in the decision process).

• You can't have it both ways --- either you want to avoid discrimination with respect to a particular characteristic, or you want to equalise process-outcomes with respect to that characteristic. If your goal is to "correct" disparities in outcomes with respect to a particular characteristic then don't kid yourself about what you are doing --- you are engaging in discrimination for the purposes affirmative action.

• Many of the more complex difficulties you raise occur when you try to model things with various types of "black box" models of the type sometimes found in machine-learning applications, where there is complexity in relation to how the variables are taken into account in the model. In these cases there are some genuine issues that arise in trying to interpret and control the way that variables are used and so the problem can become non-trivial. Nevertheless, it is useful to bear in mind that using these models is a choice made by the user. If control over use of variables is important then it is better to use traditional statistical models (e.g., regression, GLMs, etc.) where there is greater clarity as to how variables are used in predictions and decisions.

Once you understand these basic aspects of statistical decision-making processes, you will be able to formulate what your actual goal is in this case. In particular, you will need to decide whether you want a non-discriminatory process, which is likely to result in disparities of outcome across groups, or whether you want a discriminatory process designed to yield equal process outcomes (or something close to this). Ethically, this issue mimics the debate over non-discrimination versus affirmative-action.

---

Let's say I want to build a statistical model to predict some output from personal data, like a five star ranking to help recruiting new people. Let's say I also want to avoid gender discrimination, as an ethical constraint. Given two strictly equal profile apart from the gender, the output of the model should be the same.

It is easy to ensure that the ratings given from the model are not affected by a variable you want to exclude (e.g., gender). To do this, all you need to do is to remove this variable as a predictor in the model, so that it is not used in the rating decision. This will ensure that two profiles that are strictly equal, apart from that variable, are treated the same. However, it will not necessarily ensure that the model does not discriminate on the basis of another variable that is correlated with the excluded variable, and it will not generally lead to outcomes that are equal between genders. This is because gender is correlated with many other characteristics that might be used as predictive variables in your model, so we would generally expect outcomes to be unequal even in the absence of discrimination.

In regard to this issue, it is useful to demarcate between characteristics that are inherent gender characteristics (e.g., pees standing up) versus characteristics that are merely correlated with gender (e.g., has an engineering degree). If you wish to avoid gender discrimination, this would usually entail removing gender as a predictor, and also removing any other characteristic that you consider to be an inherent gender characteristic. For example, if it happened to be the case that job applicants specify whether they pee standing up or sitting down, then that is a characteristic that is not strictly equivalent to gender, but one option effectively determines gender, so you would probably remove that characteristic as a predictor in the model.

Again, here we also need to note that certain complex machine-learning models might have a "black box" element where it is more difficult to understand and control the use of variables in prediction and decisions. Using these latter models is a choice, so if you value control highly (e.g., to constrain the use of variables in accordance with some ethical principle) then it is best to eschew these types of models and use traditional statistical models where it is simple to understand and control the use of input variables.

1. Should I use the gender (or any data correlated to it) as an input and try to correct their effect, or avoid to use these data?

Correct what exactly? When you say "correct their effect" I am going to assume that you mean that you are considering "correcting" disparities in outcomes that are caused by predictors that are correlated with gender. If that is the case, and you use gender to try to correct an outcome disparity then you are effectively engaging in affirmative action --- i.e., you are programming your model to discriminate positively on gender, with a view to bringing the outcomes closer together. Whether you want to do this depends on your ethical goal in the model (avoiding discrimination vs. obtaining equal outcomes).

2. How do I check the absence of discrimination against gender?

If you are talking about actual discrimination, as opposed to mere disparities in outcome, this is easy to constrain and check. All you need to do is to formulate your model in such a way that it does not use gender (and inherent gender characteristics) as predictors. Computers cannot make decisions on the basis of characteristics that you do not input into their model, so if you have control over this it should be quite simple to check the absence of discrimination. One way you could test this is to take a new test data point (e.g., a new applicant resume) and create two versions that flip the gender of the applicant, then input them into your model to make predictions/decisions --- if the model is operating in a non-discriminatory way then it should make the same prediction/decision for the new data point irrespective of whether you flip the gender.

Things become a bit harder when you use machine-learning models that try to figure out the relevant characteristics themselves, without your input. Even in this case, it should be possible for you to program your model so that it excludes predictors that you specify to be removed (e.g., gender), and it is certainly possible to test for discrimination using test data.

3. How do I correct my model for data that are statistically discriminant but I don't want to be for ethical reasons?

When you refer to "statistically discriminant" data, I assume that you just mean characteristics that are correlated with gender. If you don't want these other characteristics there then you should simply remove them as predictors in the model. However, you should bear in mind that it is likely that many important characteristics will be correlated with gender. Any binary characteristic will be correlated with gender in any case when the proportion of males with that characteristic is different from the proportion of females with that characteristic. (Of course, if those proportions are close you might find that they difference is not "statistically significant".) For more general variables the condition for non-zero correlation is also very weak. Thus, if you remove all characteristics that show evidence of non-zero correlation with gender, you will almost certainly remove a number of important predictors, and you will not have much left.

Answer 6

This at most will be a partial answer (or no answer at all).

First thing to note is that I agree with @dsaxton completely: all models "discriminate" (at least in some definitions of discrimination) as that is their function. The issue is that models work on summaries and averages and they assign things based on averages. Single individuals are unique and might be completely off the prediction.

Example: consider a simple model that predicts the mentioned five star ranking based on one variable - age. For all people with the same age (say 30) it will produce the same output. However that is a generalisation. Not every person aged 30yr will be the same. And if the model produces different ranks for different ages - it is already discriminating people for their age. Say it gives a rank of 3 for 50 year olds and a rank of 4 for 40 year olds. In reality there will be many 50 year old people that are better at what they do than 40 year old. And they will be discriminated against.

---

1. Should I use the gender (or any data correlated to it) as an input and try to correct their effect, or avoid to use these data?

If you want the model to return the same outcome for otherwise equal men and women then you should not include gender in the model. Any data correlated to gender should probably be included. By excluding such covariates you can be making at least 2 types of errors: 1) assuming all men and women are equally distributed across all covariates; 2) if some of those gender-correlated covariates are both relevant to the rating and correlated with gender at the same time - you might vastly reduce the performance of your model by excluding them.

2. How do I check the absence of discrimination against gender?

Run the model on exactly the same data twice - one time using "male" and another time using "female". If this comes from a text document maybe some words could be substituted.

3. How do I correct my model for data that are statistically discriminant but I don't want to be for ethical reasons?

Depends on what you want to do. One brutal way to force equality between genders is to run the model on men applicants and women applicants separately. And then choose 50% from one group and 50% from another group.

Your prediction will most likely suffer - as it is unlikely the best set of applicants will include exactly half men and half women. But you would probably be OK ethically? - again this depends on the ethics. I could see an ethical declaration where this type of practice would be illegal as it would also discriminate based on gender but in another way.

Answer 7

1. Should I use the gender (or any data correlated to it) as an input and try to correct their effect, or avoid to use these data?

There are several implications of this question that boil down to the following, Do I want to be a social engineer; an activist whose role is to change the status quo because I have decided that society is sick and requires therapy? The obvious answer to this depends on whether or not such a change is beneficial or harmful. For example, the answer to "What would we gain from gender equality for nursing staff?" might be that having at least one male nurse available for inserting urinary catheters in males would not require that as many as 50% of nurses be male. So, the social engineering approach examines different cultures, contexts and problems with known gender bias, and posits functional benefits to be had from alterations of the root cause(s) of that bias. This is an essential step in the decision making process. Now, the answer to question 1. is a resounding no, that is, once one has decided that society needs fixing, one just adds a star, or fraction there of (see below), to female applicants, but be very careful of what you wish for because this is affirmative action, which is itself inherently discriminatory. Any AI outcomes will change to reflect the new hiring norms, once those become established as a new functional norm.

2. How do I check the absence of discrimination against gender?

Simple enough, after ratings are assigned, one does a post hoc analysis to see what the distribution of ratings are for males and female and compare them.

3. How do I correct my model for data that are statistically discriminant but I don't want to be for ethical reasons?

This is unavoidably done after the fact, i.e., post hoc. Forethought is also necessary, but the type of forethought most needed is a concerted attempt to examine critically what the social engineer's assumptions are. That is, assuming (for the sake of argument, see below) it to be sociologically justifiable to eliminate all gender bias, one merely adjusts the female ratings to follow the same empirical distribution as the males. In the teaching business this would be called grading on a curve. Further, let us suppose that it may not be desirable to do a full elimination of gender bias (it may be too disruptive to do so), then one can do a partial elimination of bias, e.g., a pairwise weighted average of each native female rating and its fully corrected rating, with whatever weights one wishes to assign that is thought (or tested as being) least harmful and/or most beneficial.

Gender disparity cannot be altered properly by hiring policies alone as in some fields there is a relative scarcity of women candidates. For example, in Poland, 14.3% of IT students were female in 2018, and in Australia 17%. Once hired, retention of women in tech-intensive industries was problematic (Women in business roles in tech-intensive industries leave for other industries at high rates—53% of women, compared to 31% of men.) Thus, female job satisfaction may be more important than hiring policy alone. One first needs to identify a tangible benefit for having any particular percentage of females in the work place, and there are some hints about this, for example, in 2016, women on corporate boards (16%) were almost twice as likely as their male counterparts (9%) to have professional technology experience among 518 Forbes Global 2000 companies. Thus tech-savviness appears to contribute more to female than male net worth. From this discussion, it should be obvious that before making gender specific assumptions, a substantial effort should be directed toward identifying more global concrete benefits of specific policies of which hiring policy is only a small, albeit important, part, and probably not the most important starting point. That latter is plausibly the retention of hires because turnover is bad for moral and may be the root cause of gender bias in hiring.

My management experience has taught me that even small changes in work output (e.g. 10-20%) are quite effective in eventually eliminating wait lists, that is, there is no need to immediately increase output 100% by doubling staff numbers as the effect of that will shorten the wait list only slightly faster than a smaller change will, but will then be disruptive as staff will subsequently be standing around hoping that work will walk in the door. That is, if one decides to do social engineering, it can be harmful to attempt a full correction; it doesn't work that way. Try that with an abrupt course correction in a sailboat, and one may wind up exercising one's swimming lessons. The equivalent for treating gender bias (if the prescription fits), would be to only hire females. That would solve the problem (and create others). So, my advice would be to gradually correct any perceived (better would be demonstrated) problem (e.g., female job retention), to subsequently reorient to see the effect of any policy change, and adjust as needed thereafter.

In summary, effectual social engineering requires a holistic approach to complicated situations, and merely identifying that there may be a problem does not tell us there is one, does not tell us what causes it, does not tell us how to correct it, and indeed all it tells us is that we have to put on our thinking caps.