I'm working on a Regression problem consisting on predict some continuous variable $Y$, as usual. The thing here is that there are two kinds of data examples: some that have a value of $Y$ a little larger than they should, and some that have it exactly right, so in the optimization task of the algorithm (min. the loss function) the magnitude of $Y$ should be considered, as not only is important to get as close to $Y$ as possible, but as low $Y$ is, the better.
For example, in a regular regression problem, in the case below the grey line would be our best possible model because the error is minimum for it, but for this case not only the error is important, the magnitude of the output matters too, so imagine that A point has a quite lower $Y$ than B, it will provoke the model to shift towards it, as seen in the red line, because it minimizes the error and the magnitude.
My question here is how this could be achieved? Is there any technique to weight the examples before learn to predict them, so receive more importance those with lower $Y$? Is it possible to add this magnitude to the loss function in order to try to minimize it too?
I don't know exactly how to think about this, so whatever help you could provide would be quite useful. Thanks!
