2
$\begingroup$

I've been asked to fit a ZeroInflatedPoisson model on a dataset to predict Y (count data) for an assignment. First, I did this manually:

  1. Create a binary variable (Y_IND) based on Y where Y_IND = 0 if Y = 0, and 1 if Y >=1.
  2. Fit a statsmodels Logistic Regression model using X variables to predict the binary variable Y_IND with no problem.
  3. Fit a statsmodel Poisson Regression model on the subset of data where Y_IND = 1, using X variables to predict Y, which also worked without issue.

However, when I tried to fit a model using statsmodels ZeroInflatedPoisson to predict Y, I received error messages stating: "ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals ConvergenceWarning)"

This is my code: sm.ZeroInflatedPoisson(endog=y, exog=X, exog_infl=X, inflation='logit').fit(maxiter=100)

I assume the warning message is referring to the Logistic Regression step, but I'm not sure why since it was able to fit a Logistic Regression model independently on the same dataset.

Is it unable to converge because it is trying predict Y instead of the binary Y_IND? If so, how can I get around this issue?

Improve this question
$\endgroup$
3
  • $\begingroup$ Can you make the data available? Do you actually have excess zeros in the data? Zero-inflated models will not converge or fail if there is a zero-deficit instead of excess. Try model.fit(method="nm", maxiter=5000) as a more robust optimizer. $\endgroup$
    – Josef
    Mar 7 at 13:09
  • $\begingroup$ @Josef Thanks for your reply. Here is a link to the data if you are still interested. link $\endgroup$
    – Akimon
    Mar 8 at 16:15
  • $\begingroup$ The response variable is STRESS. $\endgroup$
    – Akimon
    Mar 8 at 16:15

1 Answer 1

Reset to default
2
$\begingroup$

In this case the warning is noise and can be ignored. The final model converges and the results look good. The warning comes from estimating a simplified model for starting parameters which is not run until convergence (I had forgotten to silence it.).

In general models that are strongly misspecified for the given data will often have convergence problems.
This is the case in Zero-inflated models if there is no zero inflation or even a deficit of zeros.
A similar case, NegativeBinomial is unlikely to converge if there is no overdispersion relative to Poisson.

In those cases it is better to estimate the basic model first and check that the deviation or misspecification is in the direction of the alternative model.

The following illustrates that for zero-inflation.

rhs = 'AGE + COHES + ESTEEM + GRADES + SATTACH'
mod = Poisson.from_formula("STRESS ~" + rhs, data)
res = mod.fit()
diag = res.get_diagnostic()

The hypothesis test that there is no zero-inflation (or deficit) is strongly rejected:

diag.test_poisson_zeroinflation().pvalue
# 3.9959363587290166e-33

As a quick diagnostic for how well the model fits the data, we can compare the observed frequencies of count with the expected probabilities for each count, both averaged over the estimation sample.
We can see that more zeros are observed than predicted.

diag.plot_probs(),

Poisson plot_probs

So, estimating the Poisson model indicates that there is an excess of zeros.

Next, estimate the zero-inflated Poisson model.
It shows that the estimation converged. Comparing observed and expected frequencies shows that the ZIP model fits the data much better than the Poisson model.

mod_zip = cm.ZeroInflatedPoisson.from_formula("STRESS ~" + rhs, data)
res_zip = mod_zip.fit()

res_zip.converged
# True

diag_zip = res_zip.get_diagnostic()

diag_zip.plot_probs(),

ZIP plot_probs

(Note: get_diagnostic will be available in the next statsmodels release)

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you very much for the demonstration and explanation. In regards to the graph (third from bottom), is the orange line showing the frequencies of predicated values? $\endgroup$
    – Akimon
    Mar 10 at 1:49
  • $\begingroup$ orange line is predicted distribution, i.e. sum over observation i of zip.pmf(count| xi) for counts on horizontal axis. Default predicted values are expectation E(y |xi) which is not the distribution of counts for a new observation. $\endgroup$
    – Josef
    Mar 10 at 13:17
  • $\begingroup$ for some reason when I made predictions using the zip model it did not produce any 0s. Do you think I'm doing something wrong? $\endgroup$
    – Akimon
    Mar 12 at 22:14
  • $\begingroup$ predict() estimates the mean conditional on exog. That will not be zero (except in degenerate cases). You can look at predict(which=???) for the different predicted statistics, e.g. probability of getting a zero. $\endgroup$
    – Josef
    Mar 13 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.