I have questions inspired from the Amazon recent recruitment scandal, where they were accused of discrimination against women in their recruitment process. More info here:

Amazon.com Inc's machine-learning specialists uncovered a big problem: their new recruiting engine did not like women.
The team had been building computer programs since 2014 to review job applicants' resumes with the aim of mechanizing the search for top talent...
... The company's experimental hiring tool used artificial intelligence to give job candidates scores ranging from one to five stars...
... But by 2015, the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way.
That is because Amazon's computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry. (For a graphic on gender breakdowns in tech, see: here) In effect, Amazon's system taught itself that male candidates were preferable. It penalized resumes that included the word "women's," as in "women's chess club captain." And it downgraded graduates of two all-women's colleges, according to people familiar with the matter. They did not specify the names of the schools.
Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said.
The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project...
... The company's experiment... offers a case study in the limitations of machine learning.
... computer scientists such as Nihar Shah, who teaches machine learning at Carnegie Mellon University, say there is still much work to do.
"How to ensure that the algorithm is fair, how to make sure the algorithm is really interpretable and explainable - that's still quite far off," he said.

[Amazon] set up a team in Amazon's Edinburgh engineering hub that grew to around a dozen people. Their goal was to develop AI that could rapidly crawl the web and spot candidates worth recruiting, the people familiar with the matter said.
The group created 500 computer models focused on specific job functions and locations. They taught each to recognize some 50,000 terms that showed up on past candidates' resumes. The algorithms learned to assign little significance to skills that were common across IT applicants, such as the ability to write various computer codes...
Instead, the technology favored candidates who described themselves using verbs more commonly found on male engineers’ resumes, such as "executed" and "captured," one person said.

Let's say I want to build a statistical model to predict some output from personal data, like a five star ranking to help recruiting new people. Let's say I also want to avoid gender discrimination, as an ethical constraint. Given two strictly equal profile apart from the gender, the output of the model should be the same.

  1. Should I use the gender (or any data correlated to it) as an input and try to correct their effect, or avoid to use these data?

  2. How do I check the absence of discrimination against gender?

  3. How do I correct my model for data that are statistically discriminant but I don't want to be for ethical reasons?

  • 1
    $\begingroup$ I think a reference to an article about Amazon's supposed recruitment scandal would be important to have in the body of you question. Some could argue there was no "discrimination" (depending on how the term is defined) at all but a simple imbalance in men vs. women hired, so maybe provide your definition of "discrimination." $\endgroup$ – StatsStudent Jan 8 '19 at 1:30
  • 1
    $\begingroup$ What about the possibility that a canonical answer is not possible here? Can you address that? I'm considering reclosing. $\endgroup$ – gung - Reinstate Monica Jan 13 '19 at 16:55
  • 3
    $\begingroup$ "absence of impact of the gender on the output of the model" - I don't see why is this a good definition. Suppose, that you don't have a feature called gender in the model, which it probably doesn't anyway. Then model "figures out" by looking at other attributed that it's a woman, and downgrades because a woman is less fit for the job then what? There are many jobs where women would be poor fit, e.g. marines. It doesn't mean you can;t find a woman that would be great, but most of them won't fit. $\endgroup$ – Aksakal Jan 16 '19 at 3:26
  • 2
    $\begingroup$ @Lcrmorin, why only physical? Why do you think the programmer is a best fit for both women and men? I don't see the obvious reason. I'm suspicious because only high paid jobs are in question. For instance ,men get jailed more, why not push to get women jailed at the same rate? A good definition of what constitutes a discrimination is paramount $\endgroup$ – Aksakal Jan 21 '19 at 0:59
  • 1
    $\begingroup$ I think this is covered by any applied statistics course in social sciences trying to determine causal effects... Identifying omitted variable bias etc, ' controlling for factor X' etc $\endgroup$ – seanv507 Jan 22 '19 at 7:04

This paper provides an excellent overview of how to navigate gender bias especially in language-based models: Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings - Bolukbasi et. al.. A nice blog summary can be found here:


You'll find a larger compendium of resources here:


You'll find a slew of techniques in the above links to mitigate gender bias. Generally speaking they fall into three classes:

1) Under/Over sampling your data. This is intended to oversample high-quality female resumes and under sample male resumes.

2) Subtracting out the "gender subspace." If your model is gender-biased, then you could demonstrate it to be so by using your resume embeddings to directly predict gender. After building such an auxiliary model (even just sampling common terms belonging to either gender, and then applying PCA), you can in effect subtract out this dimension from the model, normalizing the resume to be gender-neutral. This is the main technique used in Bolukbasi's paper.

3) Adversarial Learning. In this case you try to generate additional data by trying to generate more versions of high quality female resumes that are otherwise indistinguishable from real resumes.

  • 1
    $\begingroup$ No offense here but something is missing. Is this treating the symptoms while ignoring the disease? Is a PC language solution a solution at all or is it problem in it's own right? Language is not the primary source of feelings, only a means of expressing them. Feelings are reactions to situations. Fix the situations, rather than paper them over with language. $\endgroup$ – Carl Jan 22 '19 at 6:06
  • $\begingroup$ @Carl: If a dataset is rotten with bias already, it’s very difficult to correct. Political correctness is an attempt at mitigating bias. In the ML case we are guiding the model away from biasing it’s predictions based on gender to those that are purely based on skill. Otherwise such a model will likely penalize females and assign very different scorings to their skills at each profession. Read Dave Harris’s answer for additional ways bias arises and how to fix them through physical changes (as opposed to data changes) $\endgroup$ – Alex R. Jan 22 '19 at 7:34
  • $\begingroup$ I commented on Dave's answer, so your suggestion shows you missed that. You may also be missing the point that to win a job you hate is a problem. Fix the retention problem, and the working environment. Making females more "attractive" solves nothing, it can exacerbate the problems. With respect to a job, the problem is not "getting married" to it, but "staying married". $\endgroup$ – Carl Jan 22 '19 at 12:46
  • $\begingroup$ @Carl: I'm not sure what you're arguing here, as OPs question is clearly asking about how to build a statistical model on an existing dataset. The links I provided show that language models, out-of-the-box, can already contain hidden biases. I could just as well argue that people who keep their jobs long enough are likely too mediocre to find jobs elsewhere. Regardless of what KPI you're optimizing (that's a relevant but, completely separate topic), your model may still exhibit gender biases. $\endgroup$ – Alex R. Jan 22 '19 at 21:06
  • 1
    $\begingroup$ Agreed. You did answer the question. However, female job retention of Tech jobs is poor and you did not identify the problems. So the answer is a disservice to females. If used it will cause misery. Statisticians have the moral responsibility to see their work in context and identify the questions that are more appropriate than those posited naively. $\endgroup$ – Carl Jan 22 '19 at 21:59

This is not an answer to your question but just a few thoughts that are too long to fit in a comment.

I think one problem we have to consider when thinking about these issues is that every model discriminates, and they will do so on the basis of any association present in the data. That is arguably the whole purpose of a predictive model. For instance, men are genuinely more likely to commit crime than women, so almost any model that has access to this information will draw such an inference.

But that doesn't mean we should convict someone partially on the basis of gender, even though a man will generally appear more likely to have committed a crime (other things equal). Rather we should require direct evidence of a crime when making such decisions, and not information on mere association. As another example: do people who are more likely to get sick really deserve to pay higher insurance premiums?

So when it comes to discrimination, I would argue that the issue deals more with ethical application, rather than models themselves being unfair. If we are worried about perpetuating discrimination or other unfair outcomes when using a model in given situation, then perhaps we should not be using a model.

  • 2
    $\begingroup$ I would argue about your last sentence that a model built to avoid social discrimination will perform better in that matter than a human, but that's not why I am here. I will edit my post with a better definition of social discrimination. $\endgroup$ – lcrmorin Jan 8 '19 at 9:49
  • $\begingroup$ I don't think our court system intends to punish men more, but it does. The same goes for that hypothetical Amazon algorithm. I doubt that they wanted to discriminate women, but the algo learned that women are less fit and discriminated anyway. $\endgroup$ – Aksakal Jan 16 '19 at 3:29
  • $\begingroup$ You've drifted sharply from the OP's question: Contrast discrimination n 1: unfair treatment of a person or group on the basis of prejudice 2: the cognitive process whereby two or more stimuli are distinguished. [WordNet]. The OP is asking about the first definition, and you are responding about the second. $\endgroup$ – Alexis Jan 22 '19 at 2:13
  • $\begingroup$ @Alexis It is actually not clear to me that OP is only talking about the first definition. In the quote: "How do I correct my model for data that are statistically discriminant but I don't want to be for ethical reasons?" This would seem to imply that they wish to deal with things that statistically differ by gender, even if they are not traits that are unfair to use. By the way, there is also no sharp distinction between the two concepts of discrimination. What some people find fair others find unfair. $\endgroup$ – Eff Jan 22 '19 at 9:00
  • $\begingroup$ There are important differences with respect to gender, both in term of number of people applying and in term of profile. My goal is to ensure that two people with an equivalent profile but not the same gender will be treated equally. $\endgroup$ – lcrmorin Jan 22 '19 at 10:19

I used to work on a project to develop software management best practices. I observed roughly fifty software teams in the field. Our sample was around 77, but we ended up seeing around a hundred teams. In addition to collecting data on things such as certifications, degrees and so forth, we also collected a variety of psychological and demographic data.

Software development teams have some very significant self-selection effects in it that, while having nothing to do with gender, are strongly correlated with gender. Also, managers tend to replicate themselves. People hire people they are comfortable with, and they are most comfortable with themselves. There is also evidence that people are being rated in a cognitively biased way. Imagine that, as a manager, I highly value prompt arrival at the start of work. I will then rate on that. Another manager, who just cares that the work gets done, may rate on something entirely different as important.

You noted that men use language differently, but it is also true that people with different personalities use language in different ways. There may be ethnic language usage differences as well, see for example the current controversy at Harvard and Asian admissions.

Now you assume that the software firms discriminate against women, but there is another form of gender discrimination going on in the software development industry that you haven’t accounted for. When you control for objective things such as certifications, degrees, tenure and so forth, the average woman earns 40% more than the average man. There are three sources of employment discrimination in the world.

The first is that managers or owners do not wish to hire someone on the basis of some feature. The second is that coworkers do not wish to work with the people with that feature. The third is that customers do not want people who have a feature. It appears the wage discrimination is being triggered by customers because the work product is different, and from the customers’ perspectives, also better. This same feature causes male dental hygienists to take lower pay than women. It is also seen in a bias toward “born here” in world soccer wages.

The best control for this is to understand your data and the social forces involved. Any firm that uses its own data will tend to replicate itself. That may be a very good thing, but it could also make them blind to forces at work. The second control is to understand your objective function. Profits may be a good function, but it may be a bad function. There are values in play in the selection of an objective loss function. Then, finally, there is the issue of testing the data against demographics to determine if unfortunate discrimination is happening.

Finally, and this is a bigger problem in things like AI where you cannot get good interpretative statistics, you will want to control for Yule’s paradox. The classic historical example is the discovery that 44% of men were accepted to UC Berkley while only 35% of women were admitted in 1973. This was a huge difference and statistically significant. It was also misleading.

This was obviously scandalous, and so the university decided to look at which were the offending majors. Well, it turned out that when you controlled for major, there was a statistically significant bias in favor of admitting women. Of the eighty-five majors, six were biased toward women and four toward men, the remainder were not significant. The difference was that women were, disproportionately, applying for the most competitive majors and so few of either gender were getting in. Men were more likely to apply to less competitive majors.

Adding in Yule’s paradox creates an even deeper layer for discrimination. Imagine, instead of a gender test, there was a gender test by type of job. You could possibly pass a company-wide gender neutral test but fail at the task level. Imagine that only women were recruited for V&V and only men for systems administration. You would look gender neutral, and you wouldn’t be.

One potential solution to this is to run competitive AIs that use differing objective criteria of “goodness.” The goal is to widen the net, not narrow it. This can also help avoid another problem in the management literature. While 3% of males are sociopaths, that number climbs substantially as you go further and further up the corporate ladder. You don’t want to be filtering for sociopaths.

Finally, you may not want to consider using AI for certain types of positions. I am job hunting right now. I am also sure I am being filtered out, and I haven’t figured out how to get around it. I am sitting on a very disruptive new technology. The problem is that my work doesn’t match the magic words. Instead, I have the next set of magic words. Right now, I am worth a fortune to the right firm, but in one case where I applied, I received an automated decline in less than a minute. I have a friend who has served as the CIO of federal agencies. He applied for a job where the hiring manager was waiting to see his application come through so he could pretty much be offered the job. It never came through because the filters blocked it.

This sets up the second problem of AI. If I can work out from online resumes who Amazon is hiring, then I can magic word my resume. Indeed, I am working on my resume right now to get it to fit non-human filters. I can also tell from the e-mails from recruiters that some parts of my resume are being zoomed in on and other parts ignored. It is as if the recruiting and hiring process has been taken over by software like Prolog. Logical constraints met? Yes! This is the optimal candidate or set of candidates. Are they optimal?

There isn't a pre-built answer to your question, only problems to engineer around.

  • $\begingroup$ (+1) Solid observations. I especially like the equivocation concerning interpretive bias of results and would only add that one should define a goal for social engineering that is object oriented, that is, some tangible benefit. For example, having a male nurse available for inserting urinary catheters in males would not require that 50% of nurses be male. $\endgroup$ – Carl Jan 20 '19 at 21:13
  • $\begingroup$ @Dave. thanks for your insight. Can you provide sources for the "When you control for objective things such as certifications, degrees, tenure and so forth, the average woman earns 40% more than the average man" ? and what do you mean by "You don’t want to be filtering for sociopaths." ? $\endgroup$ – lcrmorin Jan 22 '19 at 10:28
  • $\begingroup$ @Lcrmorin there is a tendency for sociopaths to seek promotion. If you are replicating your existing hierarchy because you are using its data then you could find yourself selecting for those behaviors that filter for sociopathy. There is a belief that the software will be neutral, but many existing chains of command are far from neutral. Sometimes filtering for men isn't really filtering for men, but instead, the male behaviors that disguise sociopathy. $\endgroup$ – Dave Harris Jan 22 '19 at 15:40
  • $\begingroup$ @Lcrmorin My office is literally in boxes right now as it is being moved, including about seven hundred journal articles. The article struck me because I was researching software engineers at the time. That was twelve years ago, however. Having performed field research and trained future engineers, my guess is that women have to pick up male behaviors to survive in a heavily male dominated group, but men do not have to pick up the behaviors females bring with them. My guess is that the difference enters through the requirements elicitation process. $\endgroup$ – Dave Harris Jan 22 '19 at 15:44
  • $\begingroup$ I was mainly sceptical about the 40% figure, which seems to be a lot and nowhere near what people, including me, seems to experience nowadays. $\endgroup$ – lcrmorin Jan 22 '19 at 15:52

In order to build a model of this kind, it is important to first understand some basic statistical aspects of discrimination and process-outcomes. This requires understanding of statistical processes that rate objects on the basis of characteristics. In particular, it requires understanding the relationship between use of a characteristic for decision-making purposes (i.e., discrimination) and assessment of process-outcomes with respect to said characteristic. We start by noting the following:

  • Discrimination (in its proper sense) occurs when a variable is used in the decision process, not merely when the outcome is correlated with that variable. Formally, we discriminate with respect to a variable if the decision function in the process (i.e., the rating in this case) is a function of that variable.

  • Disparities in outcome with respect to a particular variable often occur even when there is no discrimination on that variable. This occurs when other characteristics in the decision function are correlated with the excluded variable. In cases where the excluded variable is a demographic variable (e.g., gender, race, age, etc.) correlation with other characteristics is ubiquitous, so disparities in outcome across demographic groups are to be expected.

  • It is possible to try to reduce disparities in outcomes across demographic groups through affirmative-action, which is a form of discrimination. If there are disparities in process-outcomes with respect to a variable, it is possible to narrow those disparities by using the variable as a decision-variable (i.e., by discriminating on that variable) in a way that favours groups that are "underrepresented" (i.e., groups with lower proportions of positive outcomes in the decision process).

  • You can't have it both ways --- either you want to avoid discrimination with respect to a particular characteristic, or you want to equalise process-outcomes with respect to that characteristic. If your goal is to "correct" disparities in outcomes with respect to a particular characteristic then don't kid yourself about what you are doing --- you are engaging in discrimination for the purposes affirmative action.

Once you understand these basic aspects of statistical decision-making processes, you will be able to formulate what your actual goal is in this case. In particular, you will need to decide whether you want a non-discriminatory process, which is likely to result in disparities of outcome across groups, or whether you want a discriminatory process designed to yield equal process outcomes (or something close to this). Ethically, this issue mimics the debate over non-discrimination versus affirmative-action.

Let's say I want to build a statistical model to predict some output from personal data, like a five star ranking to help recruiting new people. Let's say I also want to avoid gender discrimination, as an ethical constraint. Given two strictly equal profile apart from the gender, the output of the model should be the same.

It is easy to ensure that the ratings given from the model are not affected by a variable you want to exclude (e.g., gender). To do this, all you need to do is to remove this variable as a predictor in the model, so that it is not used in the rating decision. This will ensure that two profiles that are strictly equal, apart from that variable, are treated the same. However, it will not necessarily ensure that the model does not discriminate on the basis of another variable that is correlated with the excluded variable, and it will not generally lead to outcomes that are equal between genders. This is because gender is correlated with many other characteristics that might be used as predictive variables in your model, so we would generally expect outcomes to be unequal even in the absence of discrimination.

In regard to this issue, it is useful to demarcate between characteristics that are inherent gender characteristics (e.g., pees standing up) versus characteristics that are merely correlated with gender (e.g., has an engineering degree). If you wish to avoid gender discrimination, this would usually entail removing gender as a predictor, and also removing any other characteristic that you consider to be an inherent gender characteristic. For example, if it happened to be the case that job applicants specify whether they pee standing up or sitting down, then that is a characteristic that is not strictly equivalent to gender, but one option effectively determines gender, so you would probably remove that characteristic as a predictor in the model.

  1. Should I use the gender (or any data correlated to it) as an input and try to correct their effect, or avoid to use these data?

Correct what exactly? When you say "correct their effect" I am going to assume that you mean that you are considering "correcting" disparities in outcomes that are caused by predictors that are correlated with gender. If that is the case, and you use gender to try to correct an outcome disparity then you are effectively engaging in affirmative action --- i.e., you are programming your model to discriminate positively on gender, with a view to bringing the outcomes closer together. Whether you want to do this depends on your ethical goal in the model (avoiding discrimination vs. obtaining equal outcomes).

  1. How do I check the absence of discrimination against gender?

If you are talking about actual discrimination, as opposed to mere disparities in outcome, this is easy to constrain and check. All you need to do is to formulate your model in such a way that it does not use gender (and inherent gender characteristics) as predictors. Computers cannot make decisions on the basis of characteristics that you do not input into their model, so if you have control over this it should be quite simple to check the absence of discrimination.

Things become a bit harder when you use machine-learning models that try to figure out the relevant characteristics themselves, without your input. Even in this case, it should be possible for you to program your model so that it excludes predictors that you specify to be removed (e.g., gender).

  1. How do I correct my model for data that are statistically discriminant but I don't want to be for ethical reasons?

When you refer to "statistically discriminant" data, I assume that you just mean characteristics that are correlated with gender. If you don't want these other characteristics there then you should simply remove them as predictors in the model. However, you should bear in mind that it is likely that many important characteristics will be correlated with gender. Any binary characteristic will be correlated with gender in any case when the proportion of males with that characteristic is different from the proportion of females with that characteristic. (Of course, if those proportions are close you might find that they difference is not "statistically significant".) For more general variables the condition for non-zero correlation is also very weak. Thus, if you remove all characteristics that show evidence of non-zero correlation with gender, you will almost certainly remove a number of important predictors, and you will not have much left.

  • $\begingroup$ An alternative to removing correlated variables would be to train separate models for men and women. The question then is how to use those separate models? $\endgroup$ – kjetil b halvorsen Jan 16 '19 at 9:34
  • $\begingroup$ Tedious. Whereas there is merit in forethought, for example, assumptions, e.g., "How is gender bias problematic?" no-one is all knowing and there is no substitute for checking results post hoc. $\endgroup$ – Carl Jan 20 '19 at 21:02
  • 1
    $\begingroup$ +1 for thoughtful. "All you need to do is to formulate your model in such a way that it does not use gender (and inherent gender characteristics) as predictors." That's easy to write, but beginning to create algorithms for social decisions like hiring when society is in medias res means that things like income history, educational attainment, and previous position are causally downstream of gender. $\endgroup$ – Alexis Jan 22 '19 at 2:17

This at most will be a partial answer (or no answer at all).

First thing to note is that I agree with @dsaxton completely: all models "discriminate" (at least in some definitions of discrimination) as that is their function. The issue is that models work on summaries and averages and they assign things based on averages. Single individuals are unique and might be completely off the prediction.

Example: consider a simple model that predicts the mentioned five star ranking based on one variable - age. For all people with the same age (say 30) it will produce the same output. However that is a generalisation. Not every person aged 30yr will be the same. And if the model produces different ranks for different ages - it is already discriminating people for their age. Say it gives a rank of 3 for 50 year olds and a rank of 4 for 40 year olds. In reality there will be many 50 year old people that are better at what they do than 40 year old. And they will be discriminated against.

  1. Should I use the gender (or any data correlated to it) as an input and try to correct their effect, or avoid to use these data?

If you want the model to return the same outcome for otherwise equal men and women then you should not include gender in the model. Any data correlated to gender should probably be included. By excluding such covariates you can be making at least 2 types of errors: 1) assuming all men and women are equally distributed across all covariates; 2) if some of those gender-correlated covariates are both relevant to the rating and correlated with gender at the same time - you might vastly reduce the performance of your model by excluding them.

  1. How do I check the absence of discrimination against gender?

Run the model on exactly the same data twice - one time using "male" and another time using "female". If this comes from a text document maybe some words could be substituted.

  1. How do I correct my model for data that are statistically discriminant but I don't want to be for ethical reasons?

Depends on what you want to do. One brutal way to force equality between genders is to run the model on men applicants and women applicants separately. And then choose 50% from one group and 50% from another group.

Your prediction will most likely suffer - as it is unlikely the best set of applicants will include exactly half men and half women. But you would probably be OK ethically? - again this depends on the ethics. I could see an ethical declaration where this type of practice would be illegal as it would also discriminate based on gender but in another way.

  • $\begingroup$ Why not also train separately? $\endgroup$ – kjetil b halvorsen Jan 14 '19 at 9:30
  • $\begingroup$ Would this not introduce yet another form of discrimination - men and women would be chosen according to different criteria. $\endgroup$ – Karolis Koncevičius Jan 14 '19 at 11:22
  • $\begingroup$ Maybe, but it would be worth a try. It could also make for better rules for men, not giving the machine the easy way out. $\endgroup$ – kjetil b halvorsen Jan 14 '19 at 11:25
  • $\begingroup$ "And then choose 50% from one group and 50% from another group." wouldn't that lead to positive discrimination when there is disparities in the original population (both in terms of number and profile) ? $\endgroup$ – lcrmorin Jan 22 '19 at 10:49
  • 2
    $\begingroup$ @Lcrmorin Yes of course it would. That's what they mean by "it would also discriminate based on gender but in another way." $\endgroup$ – Eff Jan 22 '19 at 11:23

What the Amazon story shows is that it is very hard to avoid the bias. I doubt that Amazon hired dumb people for this problem, or that they were lacking skills, or that they didn't have enough data, or that they didn't have enough AWS credits to train a better model. The problem was that the complicated machine learning algorithms are very good at learning patterns in the data, gender bias is exactly that kind of pattern. There was bias in the data, as the recruiters (consciously or not), favored male candidates. I'm not saying in here that Amazon is a company that discriminates job candidates, I'm sure they have thousands of anti-discriminatory policies and also hire pretty good recruiters. The problem with this kind of bias and prejudice is that exists no matter how hard you try to fight it. There are tons of psychology experiments showing that people may declare not to be biased (e.g. racist), but still make biased actions, without even realizing it. But answering your question, to have algorithm that is not biased, you would need to start with data that is free of this kind of bias. Machine learning algorithms learn to recognize and repeat the patterns they see in the data, so if your data records biased decisions, the algorithm will likely learn and amplify those bias.

Second thing is managing the data. If you want to prohibit your algorithm from learning to make biased decisions, you should remove all the information that would help if to discriminate between groups of interest (gender in here). This does not mean removing only the information about gender, but also all the information that could lead to identifying gender, and this could be lots of things. There are obvious ones like name and photo, but also indirect ones, e.g. maternal leave in resume, but also education (what if someone went to girls-only school?), or even job history (say that recruiters in your company are not biased, but what if every other recruiter before was biased, so the work history reflects all those biased decisions?), etc. As you can see, identifying those issues may be pretty complicated (another reason why Amazon may have failed).

As about questions 2. and 3., there is no easy answers and I do not feel competent enough to try answering them in detail. There is tons of literature on both prejudice and bias in society, and about algorithmic bias. This is always complicated and there is, unfortunately, no simple recipes for this. Companies, like Google, hire experts whose role is identifying and preventing this kind of bias in algorithms.

  • 1
    $\begingroup$ An alternative to removing everything that helps the model discriminate between (for concreteness) gender might be to train your model with gender, then when predicting (or whatever) run the prediction twice, once with each gender, averaging the results. $\endgroup$ – jbowman Jan 19 '19 at 0:26
  • $\begingroup$ @jbowman Resulting in little interpretive consequence, and perpetuation of in-built biases over time. $\endgroup$ – Alexis Jan 22 '19 at 2:21
  • $\begingroup$ The Amazon case does in no way conclusively show bias. It could simply be the phenomenon known as stereotype accuracy. Sometimes traits actually correlate with demographic variables. Here is an example. You know that person X is young and middle class. How likely are they to commit a violent crime? I now give you another piece of information: their sex. Does this change the likelihood? Of course. Is that bias? Of course not. It's what is known as stereotype accuracy. $\endgroup$ – Eff Jan 22 '19 at 8:42
  • 1
    $\begingroup$ @Eff and this is how discrimination happens... Females earn less on average, so let's pay them less! The whole point of not having discriminative algorithms is that you should not use such information for making decissions, even if on average it seems to work. Moreover, if often works because of the social bias (e.g. we tent to pay more to males, African Amercians are more likely to go to jail for exactly the same crime as compared to Caucasian Americans etc.), so the stereotype is accurate because there is stereotype, not because of the nature of stereotyped group. $\endgroup$ – Tim Jan 22 '19 at 9:01
  • 1
    $\begingroup$ @Tim Nope. While there can be some truth to what you're saying, by and large it is not true. I urge you to read the book "Social Perception and Social Reality: Why Accuracy Dominates Bias and Self-Fulfilling Prophecy" by Lee Jussim. In this major book the author basically reviews the entire body of scientific literature on stereotypes, bias, self-fulfilling prophecies, etc. He shows that the evidence overwhelmingly shows that what you're describing is the minority of what is occurring. $\endgroup$ – Eff Jan 22 '19 at 9:06
  1. Should I use the gender (or any data correlated to it) as an input and try to correct their effect, or avoid to use these data?

There are several implications of this question that boil down to the following, Do I want to be a social engineer; an activist whose role is to change the status quo because I have decided that society is sick and requires therapy? The obvious answer to this depends on whether or not such a change is beneficial or harmful. For example, the answer to "What would we gain from gender equality for nursing staff?" might be that having at least one male nurse available for inserting urinary catheters in males would not require that as many as 50% of nurses be male. So, the social engineering approach examines different cultures, contexts and problems with known gender bias, and posits functional benefits to be had from alterations of the root cause(s) of that bias. This is an essential step in the decision making process. Now, the answer to question 1. is a resounding no, that is, once one has decided that society needs fixing, one just adds a star, or fraction there of (see below), to female applicants, but be very careful of what you wish for because this is affirmative action, which is itself inherently discriminatory. Any AI outcomes will change to reflect the new hiring norms, once those become established as a new functional norm.

  1. How do I check the absence of discrimination against gender?

Simple enough, after ratings are assigned, one does a post hoc analysis to see what the distribution of ratings are for males and female and compare them.

  1. How do I correct my model for data that are statistically discriminant but I don't want to be for ethical reasons?

This is unavoidably done after the fact, i.e., post hoc. Forethought is also necessary, but the type of forethought most needed is a concerted attempt to examine critically what the social engineer's assumptions are. That is, assuming (for the sake of argument, see below) it to be sociologically justifiable to eliminate all gender bias, one merely adjusts the female ratings to follow the same empirical distribution as the males. In the teaching business this would be called grading on a curve. Further, let us suppose that it may not be desirable to do a full elimination of gender bias (it may be too disruptive to do so), then one can do a partial elimination of bias, e.g., a pairwise weighted average of each native female rating and its fully corrected rating, with whatever weights one wishes to assign that is thought (or tested as being) least harmful and/or most beneficial.

Gender disparity cannot be altered properly by hiring policies alone as in some fields there is a relative scarcity of women candidates. For example, in Poland, 14.3% of IT students were female in 2018, and in Australia 17%. Once hired, retention of women in tech-intensive industries was problematic (Women in business roles in tech-intensive industries leave for other industries at high rates—53% of women, compared to 31% of men.) Thus, female job satisfaction may be more important than hiring policy alone. One first needs to identify a tangible benefit for having any particular percentage of females in the work place, and there are some hints about this, for example, in 2016, women on corporate boards (16%) were almost twice as likely as their male counterparts (9%) to have professional technology experience among 518 Forbes Global 2000 companies. Thus tech-savviness appears to contribute more to female than male net worth. From this discussion, it should be obvious that before making gender specific assumptions, a substantial effort should be directed toward identifying more global concrete benefits of specific policies of which hiring policy is only a small, albeit important, part, and probably not the most important starting point. That latter is plausibly the retention of hires because turnover is bad for moral and may be the root cause of gender bias in hiring.

My management experience has taught me that even small changes in work output (e.g. 10-20%) are quite effective in eventually eliminating wait lists, that is, there is no need to immediately increase output 100% by doubling staff numbers as the effect of that will shorten the wait list only slightly faster than a smaller change will, but will then be disruptive as staff will subsequently be standing around hoping that work will walk in the door. That is, if one decides to do social engineering, it can be harmful to attempt a full correction; it doesn't work that way. Try that with an abrupt course correction in a sailboat, and one may wind up exercising one's swimming lessons. The equivalent for treating gender bias (if the prescription fits), would be to only hire females. That would solve the problem (and create others). So, my advice would be to gradually correct any perceived (better would be demonstrated) problem (e.g., female job retention), to subsequently reorient to see the effect of any policy change, and adjust as needed thereafter.

In summary, effectual social engineering requires a holistic approach to complicated situations, and merely identifying that there may be a problem does not tell us there is one, does not tell us what causes it, does not tell us how to correct it, and indeed all it tells us is that we have to put on our thinking caps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.