What is biased? The model, the data or both?

Question

I wonder if we look at current state-of-the-art approaches for NLU like BERT if these approaches may incorporate an inherent bias as well. It may seem quite evident that if BERT produces biased output it is simply because it's picking up the latent structure of its training input, i.e., the bias lies in the data. But then again, how do we know an architecture (in this case BERT) does not favor a structural/sociatal bias, even if we had no bias (whatever that means) in the data, which could be understood as a form of bias as well? What if BERT itself is biased, gets biased data and produces even worse results (i.e., what if both the model and data are biased)? I read this paper, which is talking about a bias in the training data. I'm trying to extend that thought a little bit and would love your view on this.

Answer 1

You don't, so each time you need to validate the results and check if they are not biased. But let's start from the fact that it is not a binary case that the model is biased, or not. If it was that simple, we would just check if the model is biased and fix it, or not release it. Biases can be subtle, for example, Google likely did extensive testing before releasing their model, but still, they missed the case where it classified dark-skin individuals as gorillas. Also, keep in mind that, as discussed by Arvind Narayanan, there are multiple definitions and measures of fairness and it is impossible to satisfy them all at the same time.

Gathering data is hard, so it often ends biased. In many cases, you have no way of obtaining big, representative samples of data, especially for data sources like text, videos, voice recordings, or images. Moreover, in some cases even if you had a sample that was random and representative, it could be biased. For example, if you gathered all the police records from a particular region, there could be systematic racial biases because of societal biases (e.g. non-white individuals having a higher chance of being arrested because of being perceived as more likely to be criminals).

Models can be biased as well. For the sake of argument, let's assume that on average males write shorter sentences than females. You made a number of decisions based on the results you were observing when training the model on this data. For example, you turned the parameters responsible for the attention span of the model. This worked well for male-generated data, because the sentences were short, but may not work for data consisting of longer sentences, where you may need to look at the broader context. The hyperparameters and other architectural decisions you made are now baked in the model, even if someone re-trains it on the new data, the biases would be there.

For more details, you can check the Practical Fairness book by Aileen Nielsen.