0
$\begingroup$

Question is in the title.

Not asking an R question, but the NaN result was in R. I just wonder why this happens for Tweedie GLMs. Example code in R, where data_train is here and data_test is here:

library(statmod)
library(tweedie)

try_tweed = glm(l2diff_spline ~ l2packagedsize * weekday * start_hrs,
                family = tweedie(var.power = 1.6),
                data = data_train)

> predict(try_tweed, data_test, type = 'response')

  [1]          NaN          NaN          NaN          NaN          NaN
  [6]          NaN  1.042680219  0.349374452  0.632474474  0.361574669
 [11]  0.386647843  0.145592504  0.304225915 16.672377569  2.657360358
 [16]  3.279574346  4.316469909  6.168233581  0.555151601  0.114290313
 [21]  0.118503223  0.123434380  0.147458853  0.151491610  0.153290405
 [26]  0.155559259  0.158065730  0.161439662  0.165863355  0.145585591
 [31]  0.147940598  0.149942523  0.152529444  0.154977386  0.157937111
 [36]  0.160695561  0.163555641  0.167152746  0.181056759  0.193456860
 [41]  0.199792332  0.203680844  0.207239801  0.210973823  0.215018959
 [46]  0.219556274  0.224477924  0.228999597  0.233378857  0.246369802
 [51]  0.264312703  0.268856778  0.273380829  0.282051560          NaN
 [56]          NaN          NaN          NaN          NaN          NaN
 [61]          NaN          NaN  0.084918793  0.116654578  0.396087908
 [66]  0.167496176          NaN  0.248896689          NaN  0.475387044
 [71]  0.317537050          NaN          NaN  0.199364143          NaN
 [76]  0.081538667  0.885886719          NaN          NaN  0.235603265
 [81]  0.120666791          NaN  0.039718305  0.006748408  0.002907689
 [86]  0.034171925  0.060948677  0.061053778  0.062730420  0.064889959
 [91]  0.080350786  0.080644379  0.084750844  0.086689737  0.090873043
 [96]  0.091875248  0.096879150  0.196647042  0.208166979  0.210759956
[101]  0.219214288  0.226268859  0.229593786  0.237279348  0.240270125
[106]  0.249043087  0.252024708  0.260065972  0.263129151  0.271886712
[111]  0.275515632

Edit: To make the data smaller, I used 50 random samples from data_train as the new training data and the first 15 observations from data_test as the new test data. I ran the same code as above and got this:

> try_tweed = glm(l2diff_spline ~ l2packagedsize * weekday * start_hrs,
                family = tweedie(var.power = 1.6),
                data = data_train)
> predict(try_tweed,
+ data_test2,
+ type = 'response')
44671 44672 44673 44674 44675 44676 44677 44678 44679 44680 44681 44682 44683 
  NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN 
44684 44685 
  NaN   NaN 

Warning message:
In predict.lm(object, newdata, se.fit, scale = 1, type = ifelse(type ==  :
  prediction from a rank-deficient fit may be misleading

I guess there's rank deficiency... I did not get that error message using the first set of (large) data. Does this mean a full rank assumption for estimating the coefficients is not met?

Improve this question
$\endgroup$
6
  • $\begingroup$ There does not seem to be a reproducible example here which it would need but perhaps better on R-help or StackOverflow? $\endgroup$
    – mdewey
    Commented Dec 9, 2016 at 12:27
  • $\begingroup$ It does sound like a statistical question, but having a simple reproducible example (with data) would rule out needless (and potentially misleading) speculation. $\endgroup$
    – whuber ♦
    Commented Dec 9, 2016 at 14:36
  • $\begingroup$ I added in data. Sorry about that. $\endgroup$
    – 193381
    Commented Dec 9, 2016 at 23:01
  • $\begingroup$ I'm sure you can strip your example down to a much smaller dataset and simpler model! In the process of doing that you are likely to discover the problem. (I suspect it's because some of the test data lie far away from the training data.) If you don't see the problem, then please post the new model and data directly. $\endgroup$
    – whuber ♦
    Commented Dec 9, 2016 at 23:12
  • 1
    $\begingroup$ Actually, I don't get NaN values with a simpler model. When the linear predictor is l2packagedsize + weekday + start_hrs, l2packagedsize * weekday + start_hrs, l2packagedsize + weekday * start_hrs, or l2packagedsize * start_hrs + weekday, there are no NaN values. There are NaN values when the linear predictor is l2packagedsize * weekday * start_hrs. $\endgroup$
    – 193381
    Commented Dec 9, 2016 at 23:16

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.