1
$\begingroup$

I've went through the tweedie chapter in the book Generalized Linear Models with Examples in R and it seems that the function tweedie.profile is used to estimate the power index for tweedie glm.

However, that function is pretty slow for even small data. I have a data set that is 60M rows so I don't think that function will work. The model I need to fit would have power index between 1 and 2 (my data has a ton of zeroes). I am not aware of any large data implementations of something like tweedie.profile. How should I go about finding such an estimate for my large data set? Do I just attempt to fit a model with various values for p and just choose one that minimizes something like MSE?

$\endgroup$
  • $\begingroup$ 1. Are you talking about the book by Dunn & Smyth? 2. "Do I just attempt to fit a model with various values for p and just choose one that minimizes something like MSE?" --- unless something strange is going on (like a borked implementation of the profile function) I'd expect that to be slower. 3. If your data have a lot of 0's, you may well find that no Tweedie model is suitable; you might be better with some zero-inflated model (which might even itself be a zero-inflated Tweedie, perhaps, but I expect you could do something simpler). 4. Are these data insurance claim payments? $\endgroup$ – Glen_b Feb 29 at 3:24
  • $\begingroup$ 1)Yes on the book 2) The reason I said that is from following an example from h2o at :docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/algo-params/… running that code on my model was very fast $\endgroup$ – RamenZzz Feb 29 at 8:27
  • $\begingroup$ 4) yes 3) currently i am just trying to recreate a prior analysis done on different software, which is why I'm focused on tweedie, but please elaborate on the idea of a zero-inflated tweedie. What is that/how is that done? $\endgroup$ – RamenZzz Feb 29 at 9:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.