12
$\begingroup$

I would greatly appreciate your advice on the following problem:

I've got a large continuous dataset with lots of zeros (~95%) and I need to find the best way to test whether certain subsets of it are "interesting", ie don't seem to be drawn from the same distribution as the rest. Zero inflation comes from the fact that each data point is based a count measurement with both true and sampling zeros, but the result is continuous as it takes into account some other parameters weighted by the count (and so if the count is zero, the result is also zero).

What would be the best way to do this? I have a feeling that Wilcoxon and even brute-force permutation tests are inadequate as they get skewed by these zeros. Focussing on non-zero measurements also removes true zeros that are extremely important. Zero-inflated models for count data are well-developed, but unsuitable for my case.

I considered fitting a Tweedie distribution to the data and then fitting a glm on response=f(subset_label). Theoretically, this seems feasible, but I'm wondering whether (a) this is overkill and (b) would still implicitly assume that all zeros are sample zeros, ie would be biased in the same way (at best) as a permutation?

Intuitively, it sounds like have some kind of hierarchical design that combines a binomial statistic based on the proportion of zeros and, say, a Wilcoxon statistic computed on non-zero values (or, better still, non-zero values supplemented with a fraction of zeros based on some prior). Sounds like a Bayesian network...

Hopefully I'm not the first one having this problem, so would be very grateful if you could point me to suitable existing techniques...

Many thanks!

Improve this question
$\endgroup$
4
  • $\begingroup$ Update. So far, I've found this paper addressing a problem similar to mine: maths.otago.ac.nz/home/downloads/david_fletcher/… $\endgroup$
    – a11msp
    Commented Mar 29, 2011 at 16:12
  • $\begingroup$ I'm wondering whether this extremely simplified approximation would make sense, given that zeros form the absolute majority: 1) find the proportion of zeros in each subset. 2) assume that in the subset with the smallest number of zeros all zeros are true. 3) from each subset, remove the proportion of zeros equal to the proportion of zeros in the most "zero-rich" dataset. 4) run standard non-parametric stats on this modified dataset. $\endgroup$
    – a11msp
    Commented Mar 29, 2011 at 17:10
  • $\begingroup$ The hyperlink to the paper in your first comment appears to be dead. Can you provide a citation instead? $\endgroup$
    – coip
    Commented Jul 5, 2019 at 20:45
  • 1
    $\begingroup$ Thanks for pointing this out : doi.org/10.1007/s10651-005-6817-1 $\endgroup$
    – a11msp
    Commented Jul 7, 2019 at 9:23

3 Answers 3

Reset to default
10
$\begingroup$

@msp, I think you are looking at a two stage model in that attachment (I did not have time to read it), but zero inflated continuous data is the type I work with a lot. To fit a parametric model to this data (to allow hypothesis tests) you can fit a two stage but then you have two models (Y is the target and X are covariates): P(Y=0 |X) and P(Y|X;Y>0). You have to use simulation to "bring" these together. Gelmans book (and the arm package in R) shows this process for this exact model (using logistic regression and ordinary linear regression with a log link).

The other option I have seen and like better is to fit a zero inflated gamma regression, which is the same as above (but gamma as the error instead of guassian) and you can bring them together for hypothesis tests on P(Y|X). I dont know how to do this in R, but you can in SAS NLMIXED. See this post, it works well.

Improve this answer
$\endgroup$
10
  • $\begingroup$ @B_Miner, thanks very much for your answer, sorry don't have enough rating to vote you up... I'll have a look at the links! My only worry about conditional models is that they postulate that zeros cannot belong to the second (continuous) component, am I right? Does my setting not feel a bit more like a mixture model? What do you think? $\endgroup$
    – a11msp
    Commented Mar 29, 2011 at 17:42
  • $\begingroup$ I've now replicated the two-stage approach proposed in the Gelman book. If subset_factor (with 25 levels) serves as subset label, the first step is fit1=glm(response~subset_factor, family=binomial); and the second step is fit2=lm(response~subset_factor, subset=response>0). I can then run simulations as they describe to obtain the distribution of fitted response values for each factor level. However, I'm still unsure how to translate this to what I need, which is (a) the probability that coefficients aren't zero and (b) significance of difference between coefficients at different factor levels. $\endgroup$
    – a11msp
    Commented Mar 29, 2011 at 19:02
  • $\begingroup$ The two stage approach (Gelman method of two seperate models) assumes two populations, those at zero and those above. $\endgroup$
    – B_Miner
    Commented Mar 29, 2011 at 19:25
  • $\begingroup$ ...so would it be appropriate to simply say that if the impact of some factor level is significant (and significantly different from that of some other factor level) in either of the two models in Gelman's method then it's significant overall? $\endgroup$
    – a11msp
    Commented Mar 29, 2011 at 19:28
  • 1
    $\begingroup$ Yes, the two stage approach (Gelman method of two seperate models) assumes two populations, those at zero and those > 0. Regarding the hypothesis tests, can you frame them in terms of the predicted values for varying levels of the inputs and construct empirical confidence intervals related to the simulations for each? For hypothesis tests for the coefficient != 0, you need to test this seperately for both models. $\endgroup$
    – B_Miner
    Commented Mar 29, 2011 at 19:32
2
$\begingroup$

A similar approach to the Fletcher paper is used in marketing testing, where we can arbitrarily separate the effects of interventions (such as advertising) into (a) a change in the number buying the brand (i.e. proportion of zeroes) and (b) a change in the frequency of buying the band (sales given sales occur at all). This is a solid approach and conceptually meaningful in the marketing context and in the ecological context Fletcher discusses. In fact, this can be extended to (c) a change in the size of each purchase.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! I'm wondering if you're aware of an existing r implementation of this? $\endgroup$
    – a11msp
    Commented Mar 30, 2011 at 7:07
1
$\begingroup$

You could treat the exact number of zeros unknown, but constrained between 0 and the observed number of zeros. This can surely be handled using a Bayesian formulation of the model. Maybe a multiple imputation method can also be tweaked to appropriately vary the weights (between 0 and 1) of the zero observations…

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.