Can a model for non-negative data with clumping at zeros (Tweedie GLM, zero-inflated GLM, etc.) predict exact zeros?

Question

A Tweedie distribution can model skewed data with a point mass at zero when the parameter $p$ (exponent in the mean-variance relationship) is between 1 and 2.

Similarly a zero-inflated (whether otherwise continuous or discrete) model may have a large number of zeros.

I'm having trouble understanding why it is that when I do prediction or calculate fitted values with these kinds of models, all of the predicted values are non-zero.

Can these models actually predict exact zeros?

For example

library(tweedie)
library(statmod)
# generate data
y <- rtweedie( 100, xi=1.3, mu=1, phi=1)  # xi=p
x <- y+rnorm( length(y), 0, 0.2)
# estimate p
out <- tweedie.profile( y~1, p.vec=seq(1.1, 1.9, length=9))
# fit glm
fit <- glm( y ~ x, family=tweedie(var.power=out$p.max, link.power=0))
# predict
pred <- predict.glm(fit, newdata=data.frame(x=x), type="response")

pred now does not contain any zeros. I thought the usefulness of models such as the Tweedie distribution comes from its ability to predict exact zeros and the continuous part.

I know that in my example the variable x is not very predictive.

Answer 1

Note that the predicted value in a GLM is a mean.

For any distribution on non-negative values, to predict a mean of 0, its distribution would have to be entirely a spike at 0.

However, with a log-link, you're never going to fit a mean of exactly zero (since that would require $\eta$ to go to $-\infty$).

So your problem isn't a problem with the Tweedie, but far more general; you'd have exactly the same issue with the ordinary Poisson (whether zero-inflated or ordinary Poisson GLM) for example, or a binomial, a 0-1 inflated beta and indeed any other distribution on the non-negative real line.

I thought the usefulness of the Tweedie distribution comes from its ability to predict exact zeros and the continuous part.

Since predicting exact zeros isn't going to occur for any distribution over non-negative values with a log-link, your thinking on this must be mistaken.

One of its attractions is that it can model exact zeros in the data, not that the mean predictions will be 0. [Of course a fitted distribution with nonzero mean can still have a probability of being exactly zero, even though the mean must exceed 0. A suitable prediction interval could well include 0, for example.]

It matters not at all that the fitted distribution includes any substantial proportion of zeros - that doesn't make the fitted mean zero (except in the limit as you go to all zeros).

Note that if you change your link function to say an identity link, it doesn't really solve your problem -- the mean of a non-negative random variable that's not all-zeros will be positive.

Answer 2

Predicting the proportion of zeros

I am the author of the statmod package and joint author of the tweedie package. Everything in your example is working correctly. The code is accounting correctly for any zeros that might be in the data.

As Glen_b and Tim have explained, the predicted mean value will never be exactly zero, unless the probability of a zero is 100%. What might be of interest though is the predicted proportion of zeros, and this can easily be extracted from the model fit as I show below.

Here is a more sensible working example. First simulate some data:

> library(statmod)
> library(tweedie)
> x <- 1:100
> mutrue <- exp(-1+x/25)
> summary(mutrue)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.3829  1.0306  2.7737  5.0287  7.4644 20.0855 
> y <- rtweedie(100, mu=mutrue, phi=1, power=1.3)
> summary(y)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.0000  0.8482  2.9249  4.7164  6.1522 24.3897 
> sum(y==0)
[1] 12

The data contains 12 zeros.

Now fit a Tweedie glm:

> fit <- glm(y ~ x, family=tweedie(var.power=1.3, link.power=0))
> summary(fit)

Call:
glm(formula = y ~ x, family = tweedie(var.power = 1.3, link.power = 0))

Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-2.71253  -0.94685  -0.07556   0.69089   1.84013  

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.816784   0.168764   -4.84 4.84e-06 ***
x            0.036748   0.002275   16.15  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for Tweedie family taken to be 0.8578628)

    Null deviance: 363.26  on 99  degrees of freedom
Residual deviance: 103.70  on 98  degrees of freedom
AIC: NA

Number of Fisher Scoring iterations: 4

Of course the regression on $x$ is highly significant. The estimated value of the dispersion $\phi$ is 0.85786.

The predicted proportion of zeros for each value of $x$ can be computed from the following formula:

> Phi <- 0.85786
> Mu <- fitted(fit)
> Power <- 1.3
> Prob.Zero <- exp(-Mu^(2-Power) / Phi / (2-Power))
> Prob.Zero[1:5]
        1         2         3         4         5 
0.3811336 0.3716732 0.3622103 0.3527512 0.3433024 
> Prob.Zero[96:100]
          96           97           98           99          100 
1.498569e-05 1.121936e-05 8.336499e-06 6.146648e-06 4.496188e-06 

So the predicted proportion of zeros varies from 38.1% at the smallest mean values down to 4.5e-6 at the largest mean values.

The formula for the probability of an exact zero can be found in Dunn & Smyth (2001) Tweedie Family Densities: Methods of Evaluation or Dunn & Smyth (2005) Series evaluation of Tweedie exponential dispersion model densities.

Answer 3

This answer was merged from another thread asking about predictions zero-inflated regression model, but it also applies to the Tweedie GLM model.

Regression-like models predict mean of some distribution (normal for linear regression, Bernoulli for logistic regression, Poisson for Poisson regression etc.). In the case of zero-inflated regression you predict mean of the zero inflated-something distribution (e.g. Poisson, binomial). When the probability density function of the non-inflated distribution is $f$, then probability density function of zero-inflated distribution is a mixture of point mass at zero and $f$:

$$ f_\text{zeroinfl}(y) = \pi \,I_{\{0\}}(y) + (1-\pi)\, f(y) $$

where $I$ is an indicator function. Zero-inflated regression model predicts mean of $f_\text{zeroinfl}(y)$, i.e.

$$ \mu_i = \pi \cdot 0 + (1-\pi)\, g^{-1}(x_i'\beta) $$

where $g^{-1}$ is an inverse of the link function. So since you are predicting the mean of this distribution, you won't see the excess zeros in your predictions since the zeros are not the mean of the distribution (while they shrink the mean towards zero), the same as linear regression does not predict the residuals.

This is illustrated on the plot below, where values of random variable $Y$ are plotted against $X$, where $Y$ follows a zero-inflated Poisson distribution with mean conditional on $X$. The black points are the actual data that were used to fit the zero-inflated Poisson regression model, the red points are the predictions, and the blue points are means of $Y$ within the six arbitrary groups of $X$ values. As you can see, clearly the zero inflated Poisson regression model estimates $E(Y|X)$.