I am using PROC GENMOD with time series data, I have tried to work with Negative Binomial, Poisson, GEE and Zero Inflated Poisson, but in each case when I score my validation dataset, I am getting predicted values which are never zero.

E.g. A customer might buy fuel every 4 weeks, the model does not predict the weeks in between fuel fills as zeros.

Is there a better way of predicting the weeks with zero fuel fills? I assumed Negative Binomial would be able to do this, but the lack of predicted zero outcomes, which represent 78% of the fuel filling weeks (by customer) has made this model ineffective.

Thanks in advance!!

  • $\begingroup$ Is there any practical value in modeling the zero values as very small, such as 1.0E-10? This might let you post-process model predictions of less than 1.0 as 0.0. $\endgroup$ – James Phillips Dec 12 '18 at 14:51
  • $\begingroup$ Amazing thanks, got a much better result, predicting 38% correctly and 78% within 15 litres of the actuals! Strange though, is it a SAS issue or a general regression issue that they cannot predict zeros? $\endgroup$ – Mo van Praag Dec 12 '18 at 15:13
  • $\begingroup$ I personally do not uses SAS. $\endgroup$ – James Phillips Dec 12 '18 at 16:18

Let $Y$ be response variable, $X$ be the covariate.

When we fit the model, we are try to find the conditional distribution of $Y$ conditional on $X$.

Here we use Poisson regression as example .

When we fit the Poisson regression, we assume that $Y$ follows Poisson distribution for the given $X$.

$$\Pr(Y=y|X=x) = \frac {e^{-\lambda(x)}\lambda(x)^y}{y!}$$ Then we assume that $\log(\lambda(x))=\beta_0+\beta_1x$. In the process of model fitting, $\beta_0$ and $\beta_1$ were estimated.

After we get the estimate of $\beta_0$ and $\beta_1$, we have the conditional distribution of $Y$. From the known distribution, we can calculate the probability of $Y=0, 1, ...$.

So model provides the conditional distribution of response variable, not the prediction of the response variable.

After model fitting, software generally also provides the mean of the response variable based on the conditional distribution. For Poisson regression, $\exp(\hat \beta_0 + \hat \beta_1x)$ is provided. But it will not be zero. This is the reason why you do not get the "predicted" values being zero.

  • $\begingroup$ Thank you! Is there any way of forcing it in SAS? Like the suggestion above to recode zeros as 1E-10? $\endgroup$ – Mo van Praag Dec 12 '18 at 17:27
  • $\begingroup$ I do not think SAS will do it. You can find a cut point for $X\beta$ such that $\Pr(Y=0|X\beta< c) > \Pr(Y=x>0|X\beta< c)$, the chance of $Y=0$ is larger than that of $Y$ be any other values, you can say the prediction of Y is 0. You just need to save $X\beta$ in SAS, and one more data step. $\endgroup$ – user158565 Dec 12 '18 at 17:38
  • $\begingroup$ Can you please simplify this comment, I don't grasp statistical symbols very well. Are you saying that if predicted Y is a small number I can assume it is zero? Or something else? Unfortunately I only have SAS, though I'd be surprised if there isn't a workable solution $\endgroup$ – Mo van Praag Dec 13 '18 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.