3
$\begingroup$

I am using Tweedie GLM as my data contains exact zeroes. However, my stats is weak and want to confirm a few things.

  1. Does Tweedie GLM assume normality of residuals?
  2. Is shapiro.test() the way for finding normality of residuals for a model with Tweedie GLM?
  3. If the data was not normally distributed nor were the residuals can I use "glht" function for post hoc analyses?

Here is the histogram of the response variable. I tried transforming the response variable but was not able to do so. Hence I have used the values as they were obtained from the dataset.enter image description here

here is the code:

require(statmod)

require(tweedie)

c0 <- tweedie.profile(y~x, data = c, p.vec = seq(1.0, 2.0, 0.01), method = "series")

c0$p.max

c1 <- glm(y~x, data = c, family = tweedie(var.power = 1.11, link.power = 0))

summary(c1)

shapiro.test(residuals(c1))



    Shapiro-Wilk normality test

data:  residuals(c1)
W = 0.81176, p-value < 2.2e-16

Residuals are not normal.

  1. Is the code correct?
  2. Is Tweedie GLM one of the options for a dataset as mine?

Any suggestions welcome. Thanks.

$\endgroup$
  • 1
    $\begingroup$ What is the physical interpretation of y? You refer to "exact zeros", what do you mean by that exactly? (Does it refer to the occurrence of a impossible event or just it was "really zero" when we measure it (e.g. $0$ goals scored in a game)) Can you please add plots of the Deviance and the Pearson residuals? Have you considered a hurdle model with zero-inflated Gaussian? (Way easier to interpreter) $\endgroup$ – usεr11852 says Reinstate Monic Aug 25 '18 at 22:52
  • $\begingroup$ Exact zeroes are what you have meant by no goals. In my case, though there was no reading for that specimen, I have considered the value zero, since it implies that some specimens were without a value. I have not tried the hurdle model. $\endgroup$ – Harshad Aug 26 '18 at 14:50
5
$\begingroup$
  1. No, a Tweedie GLM assumes that the responses follow a Tweedie distribution so, obviously, neither the data nor the ordinary residuals are expected to follow a normal distribution.
  2. No, a Shapiro test is not at all appropriate. The only practical way to examine residuals from a GLM such as this is to plot the quantile residuals. Unlike other types of residuals, the quantile residuals are normally distributed, even when y follows a mixed discrete-continuous distribution as in this case. For example, make a probability plot of the residuals:

    res <- qresiduals(c1)
    qnorm(res)
    

    The plot of residuals vs the covariate would also useful:

    plot(x, res)
    

    Note that these plots are examining whether your fitted model is appropriate as much as they are examining the distribution of y. If the second plot shows a pattern, then that would suggest you might need more or different predictors on your model.

  3. glht claims to work for any GLM, so presumably it will run on a Tweedie GLM. But there seems no reason why you need the glht function. It is easy to test the significance of your model using standard GLM functions in R:

    summary(c1)
    anova(c1, test="F")
    

    Why make the analysis more complicated than necessary?

  4. You code looks ok in principle, but obviously we can't vouch for whether your analysis is completely correct from the limited information you've given.

  5. Yes, definitely. From the limited information you've given, this seems the sort of data that Tweedie GLMs are intended for. I might change my mind if you explained the physical meaning of your data, for example what your response variable actually is and what leads to exact zeros but, from what you've said so far, the Tweedie model seems appropriate.

By the way, I assume that you have set var.power=1.11 because that was the estimate from c0$p.max.

$\endgroup$
  • $\begingroup$ The index paramter is indeed 1.11. Also I am attaching the second plot as you mentioned. The differences in the levels, B, C and G is clear? Does this mean I need to add more predictors? I have used the glht function for finding the within level differences. $\endgroup$ – Harshad Aug 26 '18 at 14:55
  • $\begingroup$ @ Gordon Smyth. Sorry I could not attach the plot. However the plot is a box plot which shows differences in the median of the three sublevels (B, C, and G) of the independent variable. Post hoc analyses using glht show significant differences in all within level comparisons. Does this indicate that the model is underfitted? $\endgroup$ – Harshad Aug 26 '18 at 15:06
2
$\begingroup$

A GLM assumes a mean relationship $$E[Y]=g(X\beta)$$ and will also assume a mean-variance relationship $$\mathrm{Var}(Y)=v(E[Y])$$ based on the distribution $p(Y|X)$. So no, a Tweedie GLM does not assume normality of residuals (variance) but assumes a power law: the spread of residuals gets larger as the mean gets larger. In constrast, a normal GLM has a constant mean-variance relationship. Therefore a better tool for picking a GLM than normality tests (which are too sensitive anyway) is looking at a mean (prediction) vs. residual plot and seeing if it follows the particular power rule of interest (here $y=\phi x^{1.11}$). Without looking at within-group plots, I cannot say much for sure, but given the natural zero inflation, Tweedie with a power greater than 1 but less than 2 seems appropriate. Other choices to consider might be a beta-regression after normalization to $(0,1)$ (you can replace 0 with machine epsilon), or even binomial.

Also, according to the documentation, glht accepts glm objects.

$\endgroup$
  • 1
    $\begingroup$ I think that OP has already estimated the power law using the tweedie.profile function, which estimates the power law by profile likelihood. Although OP doesn't say, I assume that c0$p.max is 1.11. $\endgroup$ – Gordon Smyth Aug 26 '18 at 1:38
  • $\begingroup$ Oh I see. Corrected. $\endgroup$ – deasmhumnha Aug 26 '18 at 1:56
  • $\begingroup$ The var.power (index parameter) is 1.11.What are within-group plots? $\endgroup$ – Harshad Aug 26 '18 at 14:28
  • $\begingroup$ I assumed that since you were interested in post-hoc, your independent variable is a factor with multiple levels. You should be checking if the data within each level matches a Tweedie distribution, not the aggregate data. $\endgroup$ – deasmhumnha Aug 26 '18 at 22:05
  • $\begingroup$ @Dezmond Goff, I have checked for the within-group plots. I am actually having separate models for absolute and relative sizes of the y variable. For the absolute size, all within-group levels show the Tweedie distribution. This is true for the relative size too, except that in one sub-group the values of y variable (dependent) are close to zero, (range is 0.1-0.3) but not exact zeroes. Will it be okay to use Tweedie for the relative y variable analysis too. I thought that as long as the distribution is clustered around zero one could still use Tweedie. Thanks in advance. $\endgroup$ – Harshad Aug 27 '18 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.