I am using machine learning to predict the monthly productivity (in dollars) of various groups of people. My question relates to ways of measuring the performance of my model.

The distribution of earnings is irregular, with many zeroes, as well as unusual peaks and various skews depending on the characteristics of the people I look at.

Most model metrics (e.g. RMSE, MAE) seem to focus on how well the mean is captured, but for me this is irrelevant. The average may be correct, but if the distribution of productivity is not accurate and realistic then the model is not useful.

To try and assess accuracy of the predicted distributions I have tried making my own metric where I sum the absolute difference in proportions earning each value. This is useful, but does not seem ideal - it requires some level of binning (i.e. to the nearest $) and the choice of bin is arbitrary. Also I feel that differences in different parts of the distribution should perhaps be weighted differently.

Is there an existing metric that accounts for differences between predicted and actual distributions that would be of use to me here?

  • 1
    $\begingroup$ The KL divergence comes to mind (en.wikipedia.org/wiki/Relative_entropy), would it be applicable to your distributions ? $\endgroup$ Dec 10, 2020 at 15:44
  • 1
    $\begingroup$ Thanks - it's not something I'm familiar with, so I'll look into it. It looks promising. $\endgroup$
    – rw2
    Dec 10, 2020 at 15:47
  • 1
    $\begingroup$ That's indeed a very useful tool. A symmetrical alternative is the Jensen-Shannon divergence : en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence $\endgroup$ Dec 10, 2020 at 15:51
  • $\begingroup$ A place to start is how to predict entire distributions instead of just points. Another facet of this is how to assess what the true (conditional) distribution is to which the predicted distribution would be compared. Have you figured out either of those? $\endgroup$
    – Dave
    May 31, 2023 at 20:54
  • $\begingroup$ @Dave Yes I think both of those are figured out. I'm making individual level predictions, which can be displayed as a distribution. These are compared to actuals for the same individuals, which can also be displayed as a distribution. Recently I've been using the Wasserstein metric, or Earth Movers distance to compare the distributions. $\endgroup$
    – rw2
    Jun 8, 2023 at 9:30


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.