I've made a neural network designed to do regression. However, my dataset is unbalanced, and the data in the smaller section of the dataset have very different target values than the target values in the bulk of the dataset (by orders of magnitude).
My network gets good results for the bulk of the data, but bad results in the tail. I would like to try to improve this as much as I can before I resort to making more data.
An idea is to modify the loss - ie, not use MSE. I imagine, that since the majority of the data is quite similar, this area is "swamping" the loss function. If we have two points, x=1 and y=0.001, even if when they are predicted the distance from the points are the same it can have a very different meaning. (If they are both .1 out, this will affect y "more"). Is it, therefore, sensible to write a loss function where the relative difference contributes? So instead of the loss summing over absolute differences, it sums over % differences.
It seems that MAPE is exactly what I am looking for - the loss function will treat all the data equally (though I understand there are other issues with this method). Am I correct in this assessment, and if so, are there any variants of MAPE that would be even better suited?