1
$\begingroup$

I'm building a function/package. If the users' entry data are zero inflated then the code will log-normalise them, process them, and reverse-log-normalise and bias-correct them afterwards. If they're not zero inflated it'll just process them.

Does anyone know of a test for zero inflation? Everyone knows what it IS, but the only similar question went semi-off-topic and just solved the guy's problem (How to test/prove data is zero inflated?).

Ideally I'd like something like this: ifelse(iszeroinflated(data)=true, data_to_use<-log1p(data), data_to_use<-data)

But if that doesn't exist then whatever makes logical sense to be manually put in place of "iszeroinflated". What do we think? 30% of data are zeroes? 50? More? Less? And any consideration of the shape of the probability distribution other than the zeroes? One would expect lots of non-zero low numbers as well right? And only a few high ones?

Improve this question
$\endgroup$
2
  • 4
    $\begingroup$ Not an answer to your question, but taking logs of zero inflated data 1) Will result in errors because log(0) is not defined. So you'd have to do something arbitrary. 2) Won't normalize the data. $\endgroup$
    – Peter Flom
    Aug 18, 2014 at 12:02
  • 1
    $\begingroup$ I'd just like to point out that your ifelse code is not correct syntax. It should probably be data_to_use <- if(iszeroinflated(data)) log1p(data) else data. $\endgroup$
    – Roland
    Aug 18, 2014 at 13:07

1 Answer 1

Reset to default
6
$\begingroup$

Zero-inflation is about the shape of the distribution. Therefore, you will have to specify the distribution for the non-zero part (Poisson, Negative Binomial, etc), if you want a formal test. Then you can use a likelihood ratio test to see if the zero-inflated parameters can be dropped from the model. This can be done in R.

In cruder terms, zero inflation is defined not only by proportion of zeros but also by the total number of observations. Say, if you assume a zero-inflated Poisson model and your data contain 50% of zeros, you still won't be able to say with certainty that it's zero inflated if the total number of points is only 4. On the other hand, 10% of zeros in 1000 observations can result in a positive test for zero-inflation.

Zero-inflated property is associated with count-based data, so I haven't heard of "zero-inflated normal". E.g. in this package:

cran.r-project.org/web/packages/pscl/pscl.pdf

they only consider Poisson, Negative Binomial and Geometric. What I would do is fit, say, a Poisson model where the zero and non-zero components contain only the intercept and then check if the intercept from the zero component has a significant p-value.

P.S. I also managed to find a reference to the (log) normal zero-inflated distribution, but I don't know if it's obtainable for free:

"Analysis of repeated measures data with clumping at zero", Stat Methods Med Res, August 2002

http://smm.sagepub.com/content/11/4/341.full.pdf+html

Improve this answer
$\endgroup$
2
  • $\begingroup$ Ahaa, interesting, thanks James. So for a Gaussian/normal distribution of the positives... sorry, you've lost me a little: would it be a likelihood ratio test of the probability that [all data] are normally distributed, vs [only non-zero] data? (p.s. if you know the name of that test in R off the top of your head, please say) Thanks! $\endgroup$
    – dez93_2000
    Aug 18, 2014 at 16:18
  • $\begingroup$ Cheers for the edit! $\endgroup$
    – dez93_2000
    Aug 18, 2014 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.