1
$\begingroup$

I have a dataset comprised of different ethnic groups and I want to build a classification model on this data. When I do this I find that the performance of the algorithm is better on some groups than on others, which is not desirable.

My first thought was to simply balance the ethnicities when I build my batches for the forward passes. E.g. if I have four distinct ethnic groups my in my data, and a batch size of 16 I'd just pass 4 samples from each group. This doesn't actually help much at all really. I think what's happening is the classification model is just lowering the cross-entropy on the easiest group, while letting the other groups suffer.

Is there a model that's able to optimize the 4 groups "fairly"? I would much prefer the performance of my model was quite good across all groups, than excellent at one and poor at the others.

Improve this question
$\endgroup$
10
  • $\begingroup$ How many observations do you have for each of your ethnic groups? Groups with few observations are going to be inherently harder to model accurately, all other things being equal... $\endgroup$
    – jbowman
    Commented Dec 9, 2018 at 18:30
  • $\begingroup$ It's actually pretty close - they only differ by about 5-8%. Even then, I only show the model an equal number from each ethnic group. $\endgroup$
    – Kurt Newman
    Commented Dec 9, 2018 at 18:53
  • $\begingroup$ Are you attempting to classify the targets by ethnicity or is ethnicity one of the independent variables (features) in the model, and it so happens that targets with particular ethnicities are classified with less success than others? $\endgroup$
    – jbowman
    Commented Dec 9, 2018 at 22:17
  • $\begingroup$ I'm not trying to predict ethnicity. Enthnicity could be treated as an independent variable (it's not currently). What I'm seeing is that my model "works better" on some ethnicities compared to others. $\endgroup$
    – Kurt Newman
    Commented Dec 9, 2018 at 22:18
  • $\begingroup$ What does "works better" mean? $\endgroup$
    – user2974951
    Commented Dec 10, 2018 at 7:57

1 Answer 1

Reset to default
0
$\begingroup$

Here is a detailed version of my answer.

Why do not you train one model for each ethnic group. You could also use some transfer learning. For example if your model is a neural network with five layer. You train your neural network to do the classification task. If you have 3 ethnic group (a, b, c), you make three copies of the network you trained on the whole dataset ($n_a$, $n_b$, $n_c$). For each of the network you can freeze the first four layer and train the last layer only on the ethnic group of the ethnicity you consider.

Then, each of the model would see its accuracy maximised on its ethnic group, still "knowing" about the other dataset. It is also possible that the classification problem you are facing is easier to solve (independently of the model) for some ethnicity than for some other and in this case I do not se what you can do.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for this answer. I don't really have reliable.ethnicities as a feature at test time though. $\endgroup$
    – Kurt Newman
    Commented Dec 10, 2018 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.