2
$\begingroup$

I am trying to build a GLM model (poisson family) using python statsmodels package on train data. The data I have contains categorical values as exogenous variables and numerical values for my target (endegenous variable). I did standardization for numeric values and one-hot-encoding on categorical values (drop the first level). When I fit the data into the model, I got the following exceptions :

ValueError: NaN, inf or invalid value detected in endog, 
estimation infeasible.  

The error comes when creating this model :

poisson_model = sm.GLM(endog=y_train, exog=X_train_std, 
   family=sm.families.Poisson(),
   offset = np.log(X_train_std.EXPOSITION))

The problem comes from np.log(X_train_std.EXPOSITION) since I can not apply log function on zero values. But I don't know how to correct the error. I need to take into consideration the offset and when changing its link function to identy I get EXPOSITION in the GLM output.

Any help please ? How to deal with offset that takes 0 values with a log link function ?

$\endgroup$
4
  • $\begingroup$ What do you mean by "standardization" of the numeric value, subtracting the mean and then dividing by the standard deviation? $\endgroup$
    – Dave
    Feb 9, 2022 at 17:44
  • $\begingroup$ Yes, I meant that. $\endgroup$ Feb 9, 2022 at 17:46
  • $\begingroup$ Is the response y_train always zero in the cases where X_train_std.EXPOSITION is zero? $\endgroup$ Feb 9, 2022 at 20:25
  • 2
    $\begingroup$ X_train_std.EXPOSITION is like exposure in the Poisson model. With zero exposure, there should not be any counts. In that case, dropping those observations seems appropriate. $\endgroup$
    – Josef
    Feb 9, 2022 at 20:29

1 Answer 1

1
$\begingroup$

Standardization means that you subtract the mean $\bar y$ and then divide by the standard deviation $s_y$.

$$ z_i = \dfrac{y_i - \bar y}{s_y} $$

By doing this transformation (unless all $y_i$ are equal, which not an interesting scenario), you create $z_i$ values less than zero, which are out of bounds in Poisson regression. This is what the error message is telling you.

(You also create $z_i$ values that are not even integers! Again, such values are out of bounds in Poisson regression.)

$\endgroup$
4
  • $\begingroup$ Clear enough ! Thank you for the explanantion. In this case, can I ignore standardization since I have one only numeric variable which is my offset variable ? $\endgroup$ Feb 10, 2022 at 15:39
  • $\begingroup$ I do not follow what you mean by an offset variable. @KarimaTouati $\endgroup$
    – Dave
    Feb 10, 2022 at 15:42
  • $\begingroup$ I mean weight variable // exposure.. $\endgroup$ Feb 10, 2022 at 15:43
  • $\begingroup$ Non-integers can make sense for Poisson regression. Some implementations do accept them. $\endgroup$
    – Nick Cox
    Nov 13, 2023 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.