How to handle errors in your target data?

Question

For the sake of the question, assume I'm presented with a simple classification task (for simplicity, let's assume binary classification). We are given a feature matrix $X$ and a target vector $y$ of the same length, $0, 1$ - valued as stated. My question would be how to handle errors in my target data when training some classification model (e.g. xgboost). I expect either one of two types of errors:

1. Manual entry errors, i.e. purely random errors stemming from data entry.
2. Systemic errors, example: conistent underreporting, i.e. for every $1$ in $y$ I can be sure that it was entered correctly and reflects the actual value, but occasionally an actual $1$ is entered as $0$.

I'll assume those errors appear in low frequency, but not in vanishingly small numbers (think 1%-5% errors in $y$); otherwise I would assume either ignoring the errors or abondoning the project altogether would be advised. To keep things simple, let's also assume roughly balanced classes (i.e. neither class appears less often than, say 20% in the error-free actual $y_{actual}$.).

Are there any alterations to the usual procedure and modelling that should be made here? Would the first kind of error and the second kind lead to different adjustments?

My first thought was that I probably want less variance than I would usually accept (i.e. use methods to prevent overfitting judiciously), hoping my model doesn't learn 'wrong' information.

Answer 1

Nearly always we assume that the data is noisy. That’s one of the points of thinking of data is statistics in terms of random variables. That’s also why we think of our models in form of $y = f(x) + \varepsilon$, where the $\varepsilon$ term stands for noise that cannot be modeled. Because you usually assume the data to be noisy, there’s no “extra steps” to be taken besides the standard practices of statistical modeling and machine learning. Of course, the different story is if you have additional knowledge about the nature of the noise that can be considered in the model, but this is considered in case-by-case basis.