How to estimate power index for tweedie glm on large data I've went through the tweedie chapter in the book Generalized Linear Models with Examples in R and it seems that the function tweedie.profile is used to estimate the power index for tweedie glm.  
However, that function is pretty slow for even small data.  I have a data set that is 60M rows so I don't think that function will work. The model I need to fit would have power index between 1 and 2 (my data has a ton of zeroes). I am not aware of any large data implementations of something like tweedie.profile.    How should I go about finding such an estimate for my large data set? Do I just attempt to fit a model with various values for p and just choose one that minimizes something like MSE? 
 A: General approaches to lowering the estimation time involved with tweedie.profile include:

*

*By far the biggest predictor of time is the method use to calculate the log-likliehood. The default method, inversion, is extremely long. I recommend using series instead, which in my experience, produced identical results at about 1/15th of the speed. Series is also the default method used by the mgcv package.

*Turn off do.ci to avoid calcualting the power index CI - you'd likely only be using teh estimate anyways

*Make sure do.plot=F and verbose=0 (should be default)

*Lowering the number of power values for consideration. Using 5 values means less values to fit a spline through (assuming do.smooth=T, which is default). 5 values still evaluates the usual maximum of 50 power values for the MLE of the power index (10 values considered per inserted pwer value.

An example following the above in code:
tweedie.profile(formula=your.formula, data=your.data, p.vec=seq(1.3,1.7,0.1), method='series', do.ci=F)

An alternate approach is to use software package that estimates the ideal power index automatically when fitting the model. These include cpglm and mgcv (the latter is for generalized additive models). They estimate the power index considerably more quickly, though the models themselves usually take longer to fit otherwise, so it may or may not improve computational time.
