2
$\begingroup$

I have a dataset with ~100K samples and non-negative continuous target variable. 99% of target values are zeros and the remaining 1% are right-skewed. Here are the deciles (0 and 1 correspond to min and max):

0.0       1.990
0.1      14.971
0.2      19.990
0.3      27.294
0.4      37.980
0.5      47.990
0.6      62.174
0.7      84.512
0.8     123.094
0.9     270.337
1.0    2933.610

So here's my problem. I need to choose an error function but whichever I take has certain drawbacks for my data. The dataset represents mobile apps' revenue and I believe that there are some common approaches to such data. Any ideas and recommendations will be highly helpful.

Here are my concerns:

  • MSE: I suppose that it will punish right-most values;
  • MAE: an error of 100 when predicting 2000 and when predicting 20 are intuitively two different errors and the second one should be punished more;
  • MAPE: I'd like to use MAPE but there are two serious issues:
    • it doesn't work with zero targets and I definitely need zeroes;
    • it can punish overestimation much more than underestimation and I believe that both types of errors should be treated the same;
  • MSLE (mean squared logarithmic loss): it seems to solve above-mentioned problems with MAPE but I feel it highly unintuitive: an error of 3 means that my prediction might be 3 times smaller or 3 times larger than the true value: how should I use it?

Whichever loss function I consider, I face either 99% of zeros or long right skew. Possibly I miss some function that nicely fits this situation? Or, may be< I should somehow separate the two cases and punish model for non-predicting zero in one way and for errors in predicting non-zero in some other? What are the best practices for such data?

Thanks in advance.

|cite|improve this question
$\endgroup$
  • $\begingroup$ Perhaps a two stage model would be helpful. Predict who will make a purchase, then predict spending conditional on making at least 1 purchase. $\endgroup$ – jsk Jun 7 at 2:31
  • $\begingroup$ How well is your non-zero data fit by a log normal distribution? My question comes from the age-old wisdom "first, take the log of money." It has never let me down, but there is always a first. Also, you may wish to look up zero-augmented distributions: a delta function at zero PLUS a relevant distribution above zero. Maybe this: rdrr.io/github/rmcelreath/rethinking/man/dzagamma2.html ? $\endgroup$ – Peter Leopold Jun 7 at 3:39
  • $\begingroup$ Have you considered some measure related to relative error? This would measure by what % of the actual measure your error in off, rather than the actual magnitude or that error $\endgroup$ – David Jun 7 at 7:19
  • $\begingroup$ @jsk, I indeed plan to use this two-step approach but not sure how I should evaluate the results. $\endgroup$ – Igor Jun 7 at 10:28
  • $\begingroup$ @PeterLeopold The fit is not bad. I'll check your link, thank you. $\endgroup$ – Igor Jun 7 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.