3
$\begingroup$

I've went through the tweedie chapter in the book Generalized Linear Models with Examples in R and it seems that the function tweedie.profile is used to estimate the power index for tweedie glm.

However, that function is pretty slow for even small data. I have a data set that is 60M rows so I don't think that function will work. The model I need to fit would have power index between 1 and 2 (my data has a ton of zeroes). I am not aware of any large data implementations of something like tweedie.profile. How should I go about finding such an estimate for my large data set? Do I just attempt to fit a model with various values for p and just choose one that minimizes something like MSE?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ 1. Are you talking about the book by Dunn & Smyth? 2. "Do I just attempt to fit a model with various values for p and just choose one that minimizes something like MSE?" --- unless something strange is going on (like a borked implementation of the profile function) I'd expect that to be slower. 3. If your data have a lot of 0's, you may well find that no Tweedie model is suitable; you might be better with some zero-inflated model (which might even itself be a zero-inflated Tweedie, perhaps, but I expect you could do something simpler). 4. Are these data insurance claim payments? $\endgroup$
    – Glen_b
    Feb 29, 2020 at 3:24
  • $\begingroup$ 1)Yes on the book 2) The reason I said that is from following an example from h2o at :docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/algo-params/… running that code on my model was very fast $\endgroup$
    – RamenZzz
    Feb 29, 2020 at 8:27
  • $\begingroup$ 4) yes 3) currently i am just trying to recreate a prior analysis done on different software, which is why I'm focused on tweedie, but please elaborate on the idea of a zero-inflated tweedie. What is that/how is that done? $\endgroup$
    – RamenZzz
    Feb 29, 2020 at 9:55
  • $\begingroup$ You could try the cplm package to fit a Tweedie glm using cpglm. which estimates the ideal parameters automatically by MLE much more quickly than tweedie profile (but slower than glm, will likely take minutes and maybe more with 60 million rows) $\endgroup$
    – MBorg
    Dec 8, 2021 at 13:26

1 Answer 1

Reset to default
0
$\begingroup$

General approaches to lowering the estimation time involved with tweedie.profile include:

  • By far the biggest predictor of time is the method use to calculate the log-likliehood. The default method, inversion, is extremely long. I recommend using series instead, which in my experience, produced identical results at about 1/15th of the speed. Series is also the default method used by the mgcv package.
  • Turn off do.ci to avoid calcualting the power index CI - you'd likely only be using teh estimate anyways
  • Make sure do.plot=F and verbose=0 (should be default)
  • Lowering the number of power values for consideration. Using 5 values means less values to fit a spline through (assuming do.smooth=T, which is default). 5 values still evaluates the usual maximum of 50 power values for the MLE of the power index (10 values considered per inserted pwer value.

An example following the above in code:

tweedie.profile(formula=your.formula, data=your.data, p.vec=seq(1.3,1.7,0.1), method='series', do.ci=F)

An alternate approach is to use software package that estimates the ideal power index automatically when fitting the model. These include cpglm and mgcv (the latter is for generalized additive models). They estimate the power index considerably more quickly, though the models themselves usually take longer to fit otherwise, so it may or may not improve computational time.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.