I have built a quasi-Poisson regression to predict sales of different products based on a number of explanatory variables, with an offset term for the number of days each product was on sale.

To estimate the goodness of fit of my model on test data I have followed this advice and calculated the logarithmic score using this trick of using the negative binomial distribution to account for overdispersion. Using this on my test predictions - which are for a total number of sales of each product - I get a reasonable result and can use the mean logarithmic score to compare different variations of my model.

Here is the R code I use to return the density of the quasi-poisson distribution:

dqpois <- function(n, mu, phi) {
  dnbinom(x = n, mu = mu, size = mu/(phi - 1))

which I then use to calculate the logarithmic score for each observation, given the dispersion parameter from my model sales.poisson:

test.comparison$logarithmic_score <- -log(dqpois(test.comparison$Obs, test.comparison$Pred, summary(sales.poisson)$dispersion))

I then compare my models using the mean logarithmic_score for each. So far so good (?)

Now I have been asked to compare my model with another prediction whose underlying method/model is unknown. These predictions are in the form of daily sales and are not integers.

So I don't see how to calculate the mean logarithmic score for the mystery model. I convert my observed values to daily rates, but dqpois expects an integer input for n and just returns zero for many of the observations (i.e. infinite / undefined logarithmic score). Of course, I have no idea if the model used to generate the predictions was a (quasi-)Poisson anyway, and certainly no idea what the dispersion parameter of that model is!

For now I have reverted to calculating root mean squared error and mean absolute error to compare the accuracy of my model's predictions vs. the mystery model's, even though they're not proper scoring rules. Can I do better? Can I calculate the logarithmic score (or any other proper scoring rule) without knowing which underlying distribution the model assumes??

  • $\begingroup$ Mean squared error is indeed a proper scoring rule. Even in estimating probabilities (such as logistic regression). $\endgroup$ – Josh Aug 16 '18 at 18:13
  • $\begingroup$ See someone far smarter than me fharrell.com/post/class-damage $\endgroup$ – Josh Aug 16 '18 at 18:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.