8
$\begingroup$

I analyze technical measurement data with the aim of developing a forecasting model.

The data is given as a non-negative time series (data per hour). The data looks quite wild and contain many zeros. I expect these zeros to be the result of measuring quantities that are not zero but too small. It is ok to treat those zeros as zero.enter image description here

Just transforming (Box-Cox e.g.) these data with exact zeros does not seem right. So I thought about a classical glm first. But these do not allow for mass at zero and a continuous distribution above zero.

So I stumbled upon Tweedie glms eg here. It must be quite a standard problem in the task to predict measurements. What are the pros and cons of working with Tweedie?

PS: The number of zeros decreases with the years ... I have to ask the data collector why this could be ...

Improve this question
$\endgroup$

1 Answer 1

Reset to default
10
$\begingroup$

You're right to think that a Box-Cox transformation won't deal with the zeros issue (nor indeed would any other transformation).

The Tweedie might be suitable, and is sometimes used for data like these*, but the probability of a zero is related to the $p$ (the power in the variance function).

*\ another issue to consider -- your data are observed over time, so you must also consider the possibility of time-dependence (such as autocorrelation).

A more common solution to the zeros would be a zero-inflated or hurdle model, such as a zero-inflated gamma. There are numerous questions on site on "zero-inflated"/"0-inflated" models and hurdle models.

However if your thought was correct and it's only "too small to register", that would indicate censoring.

Looking at the plot though, I have some doubts that it's an adequate explanation for what we see:

enter image description here

Between the lower two grey lines, there are only three points, but a large number of points either exactly on the line (or very close to it). That big gap would be consistent with your thought, but those three points (circled in red) do not seem consistent with it -- if those points can register, why not others?

However, such a banding feature can sometimes be seen in Tweedie distributions as well; the tricky part would be whether it's even possible to get the right mix of parameters to match both the proportion of zeros and the banding at lower values.

enter image description here

(Beware interpreting those plots; the spikes at zero are not density but probability, and strictly speaking should not be represented on the same plot as the continuous part. You can draw a cdf but it's less clear what's going on.)

However, even more seriously perhaps, the Tweedie definitely cannot reproduce the clumping behaviour at the top end of the plot (for that matter neither can any of the other models I've mentioned).

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for this very detailed answer! The other approaches that I thought of were: Heckman regression and very much related: fitting a logit on 0 or greater zero and then some glm on the values greater zero. A foercast would then first decide whether the value is zero or not and then forecast from the glm. What would be the more suitable approach? Any references to that? thank you! $\endgroup$
    – Richi W
    Jul 22, 2015 at 10:58
  • 2
    $\begingroup$ @Richard apologies for not having replied to this sooner; using a logistic regression for 0/not-0 and then a glm on the non-0 part is a common approach for data with many zeros. I can't really tell which would be more suitable for your problem; there are numerous considerations when it comes to judging suitability (including whether it would make a blind bit of sense to your intended audience; there are some audiences I'd expect would at least have heard of Tweedie GLMs but probably not zero-inflated models, others who'd have heard of 0-inflated or hurdle models but not necessarily Tweedie...) $\endgroup$
    – Glen_b
    Feb 25, 2016 at 22:29
  • 1
    $\begingroup$ If I remember correctly (but it's nearly two years ago since I generated the plots), you need smallish p, somewhere down near the Poisson, but it's not the only thing you need. Oh, hang on, the tweedie package has an example. See p13 of the documentation. cran.r-project.org/web/packages/tweedie/tweedie.pdf $\endgroup$
    – Glen_b
    Nov 10, 2017 at 10:21
  • 1
    $\begingroup$ @Glen_b Indeed. Thanks a lot. It's not just bimodality, $p=1.02$, $\phi=0.1$, $\mu=1$ (example from the documentation) yield a very weird-looking PDF with multiple local maxima. Demo: require(tweedie); tweedie.plot(seq(0,2,len=1000), power=1.02, phi=0.1, mu=1) $\endgroup$
    – amoeba
    Nov 10, 2017 at 10:56
  • 2
    $\begingroup$ Yeah, that's a good one. Some of my examples above have at least 3 modes but that's so cute I want to pet it. There's really clear banding on this one: power=1.015, phi=0.08, mu=.16 on (0,0.8) $\endgroup$
    – Glen_b
    Nov 10, 2017 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.