# Best loss function for very sparse real-valued data

Suppose the target output of my data prediction model is an $M\times N$ matrix where $95\%$ of the values are $0.0$ and the other values are anywhere between $0.0$ and $1.0$, what would be a good loss function to use for this kind of data?

As long as my model outputs a lot of $0$'s the MSE would be really small even at the start (about $10^{-3}$), and it has a hard time learning the values properly that are bigger than $0$

Any ideas? Thanks!

Can you do something with asymmetric loss, e.g. the cost of predicting zero when it should be non-zero is different from the cost of predicting non-zero when it should be zero.

• Good suggestion, asymmetric errors are under utilized in practice. Commented Jun 3, 2015 at 13:36
• I know it has been 8 years to the question and this comment but I recently ran into the same issue and this helped me! Particularly using squared difference for positive delta w.r.t target and an exponential decay for negative delta to bias towards overestimating! Commented Feb 11 at 22:32

I think the answer is probably going to depend on contextual knowledge, a loss function should always reflect your problem as well as possible and is key to having good predictions (in the "real" sense, not based on an artifical measurements).

The only help I can offer is to state that in terms of computational complexity, it's probably going to be more efficient to implement your own loss function in terms of the sparsity (especially if your matrix is big, which I suspect is the case here). One particular candidate could be a "specialized" MAD, say something like:

$Loss(X,Y) = \sum_{i,j} |X_{i,j} - Y_{i,j}|$

This can be calculated very efficiently using sparse calculations (the example they give is for multiplication, but you can modify it accordingly).

EDIT: I didn't take into account the second part of your question. For the zero problem, @jmmcd's answer is probably a good suggestion.

• Loss function must approximate the cost of the error. We often default to MSE or MAE because we're lazy :) or because they seem to correspond to to the costs reasonably well in most cases. However, whenever you have a special situation, which is clearly different from typical use cases for MSE, you have to look for a good cost function. Commented Jun 2, 2015 at 12:41

You could also consider decomposing this into two problems: first predict whether the value is non-zero or not (i.e. a binary classification problem, which could be phrased as a regression problem with a threshold, and either way could use a standard binary loss), and second for the non-zero values, predict their value (using a standard real-valued loss like RMSE, perhaps).