Suppose the target output of my data prediction model is an $M\times N$ matrix where $95\%$ of the values are $0.0$ and the other values are anywhere between $0.0$ and $1.0$, what would be a good loss function to use for this kind of data?
As long as my model outputs a lot of $0$'s the MSE would be really small even at the start (about $10^{-3}$), and it has a hard time learning the values properly that are bigger than $0$
Any ideas? Thanks!
Can you do something with asymmetric loss, e.g. the cost of predicting zero when it should be non-zero is different from the cost of predicting non-zero when it should be zero.
I think the answer is probably going to depend on contextual knowledge, a loss function should always reflect your problem as well as possible and is key to having good predictions (in the "real" sense, not based on an artifical measurements).
The only help I can offer is to state that in terms of computational complexity, it's probably going to be more efficient to implement your own loss function in terms of the sparsity (especially if your matrix is big, which I suspect is the case here). One particular candidate could be a "specialized" MAD, say something like:
$Loss(X,Y) = \sum_{i,j} |X_{i,j} - Y_{i,j}| $
This can be calculated very efficiently using sparse calculations (the example they give is for multiplication, but you can modify it accordingly).
EDIT: I didn't take into account the second part of your question. For the zero problem, @jmmcd's answer is probably a good suggestion.
You could also consider decomposing this into two problems: first predict whether the value is non-zero or not (i.e. a binary classification problem, which could be phrased as a regression problem with a threshold, and either way could use a standard binary loss), and second for the non-zero values, predict their value (using a standard real-valued loss like RMSE, perhaps).
