I have built two regression models to predict sales of different products based on a number of explanatory variables, with an offset term for the number of days each product was on sale. One is a Gaussian model, the other is a Poisson (both using a log link function) both trained on the same dataset of some 30k observations.

My initial confusion: on the training data, the Poisson regression achieved a much better fit as measured by pseudo-R², but had higher RMSE (and mean absolute error). Having read this I understand why.

My question now - when comparing goodness of fit on the test data, what's the appropriate measure to use? RMSE will necessarily favour OLS on the training data, so it doesn't seem like a fair comparison on the test data either. A logarithmic score is valid for Poisson because I have probabilities for each integer outcome but not applicable to OLS.

  • $\begingroup$ See stats.stackexchange.com/questions/142338/… $\endgroup$ – kjetil b halvorsen Oct 5 '18 at 12:53
  • $\begingroup$ Thanks @kjetilbhalvorsen but I've already adopted a log link function across both models. I'm now on to the secondary bit - which is modelling the disturbance term the right way, and what's getting closer to the mark in terms of predictions... $\endgroup$ – Tom Wagstaff Oct 5 '18 at 14:12
  • $\begingroup$ How many observations do you have? $\endgroup$ – kjetil b halvorsen Oct 5 '18 at 14:15
  • $\begingroup$ About 35k in the training set, and another 10k for test $\endgroup$ – Tom Wagstaff Oct 5 '18 at 14:38
  • $\begingroup$ So then, why not just use the test set to compare predictions with reality (well, data) and see what does best? You are not limited to just one loss function, you can 1) plot, 2) estimate bias, 3) estimate MSE, 4) estimate mean absolute error, or whatever other error metric which makes sense in your application. You can also compare loglikelihood on test data (a version of log score which makes sense for both models). $\endgroup$ – kjetil b halvorsen Oct 5 '18 at 14:46

That should be dictated by the use of the model. If used for prediction in future, choose one (or more) measures that is actually relevant for that use. R-squared are typically not good measures for this use, MSE (RMSE) or MAE are typically better, and you can compute both.

You can not use AIC/BIC or such in this situation, as the likelihoods are defined with respect to different dominating measures (Lebesgue measure, count measure).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.