Add training data to fix prediction problems

Question

Let's suppose I have a classification problem with three y classes {bad, middle, good}.

Data in my dataset and also data in true reality are identically distributed between the three labels (33%, 33%, 33%).

My dataset has 10000 record, so I use 8000 record for training and 2000 for test and at the end I get my model.

After deploy, I notice that {bad} and {middle} predictions are accurate, while {good} are not accurate at all. So I take other 3000 new data ONLY of {good} to add to my model and want to re-train. I split also my 3000 in 80% (2400) and 20% (600).

I have machine power, so no problem to re-train everything from the very beginning, if necessary.

My question is: even if I re-train from the very beginning my full 8000+2400 train record dataset, I have now a distribution of my records that does NOT reflect reality: reality is (33%, 33%, 33%), while my dataset with new 2400 {good} data is distributed like (60%, 20%, 20%), so, it is unbalanced versus {good}.

Am I risking to bias something, or it is fine anyway? And why?

Answer 1

The reality is reflected by your testing set. As long as the distribution of test set is close to reality then you are you mimicking testing on the reality, as the test set is the set of observations which were not seen by the model.

Answer 2

Adding more training data for just one class will add bias to your model. The imbalance in the training data will mean the model puts more importance on getting correct predictions of the {good} class at the expense of the other classes. So while this should mean your model performs better on the {good} class, it may also perform worse on the other classes. Whether or not this is a problem depends on how much the results for each class change and the relative importance of the classes. So try it and see what happens - you can always go back to your original model if necessary. I suggest adding all the new data to your training data, though. If you add some of it to the test data so it no longer reflects reality, results from your test data will also not reflect performance on "real" data.