2
$\begingroup$

I have a dataset with social media posts and like to predict the number of likes a post receives. So I fit a Generalised Linear Model (GLM). I am relatively new to GLMs but find them super cool. Inspecting the data, I think that they are overdispersed, but am not so sure about zero-inflation. This stack exchange post differentiates the two concepts. Here is my approach to check for overdispersion and zero-inflation.

The dependent variable number of likes ranges from 0 to 37,659. Out of the 8000 posts, 4359 posts (or roughly 54%) receive no likes. 3641 posts (46%) received at least one or more likes. The majority of posts that receive a like receive around 5 likes.

Comparing these numbers to a simulated quasi-poisson distribution based on this tutorial for R for 8000 data points, the range of number of likes should be 0 to 43 likes. Out of 8000 data points, 2415 (or 30%) should receive a like and 5585 (70%) should not receive a like.

Here a basic plot comparing both distributions. Histogram count data distribution

I read around a bit what the options are for overdispersion and zero-inflation.

--> which one to use?
--> From what I understand the glm() function with family set to 'quasi-poission' in R automatically corrects for the degree of overdispersion?
--> What if a comparison of both models show a similar root mean-square error?

  • Zero-inflated Poisson (ZIP) deal with excessive zeros, but I am not sure that this is the problem here? Also, I have no a priori rationale for the two processes that underlie ZIPs.

Any thoughts highly appreciated.

Fitting a GLM with a quasi-poisson distribution gives me results that I can "sell", but I want to make sure I pick a robust model that fits the data well.

Improve this question
$\endgroup$
8
  • $\begingroup$ Have you considered an ordinal regression model? $\endgroup$
    – Noah
    Commented Aug 22, 2023 at 15:57
  • 3
    $\begingroup$ if you want to deal simultaneously with zero inflation and overdispersion, something to try would be a hurdle model (search.r-project.org/CRAN/refmans/pscl/html/hurdle.html). What's nice about these is they try to explain why you're seeing zeros (or nonzeros). $\endgroup$
    – John Madden
    Commented Aug 22, 2023 at 16:02
  • $\begingroup$ You aren't going to find any (meaningful) distribution that has a 54% probability of zero and a 1/10,000 probability of > 30,000; that's a staggering amount of skew. I'd probably model it in two stages: P(like = 0) vs P(like > 0) and then P(like | like >= 1). A quick experiment with the zeta distribution on the latter indicates a parameter in the vicinity of 1.7 would give you a 1/4000 chance of an observation > 30,000, but that hardly qualifies as anything but "extremely rough cut, almost to the point of worthless" analysis. $\endgroup$
    – jbowman
    Commented Aug 22, 2023 at 17:18
  • $\begingroup$ @jbowman Just make a mixture of a dirac delta distribution with a uniform distribution; nicely characterizes the "L" shape of the data ;-) $\endgroup$
    – Galen
    Commented Aug 22, 2023 at 19:32
  • 1
    $\begingroup$ hurdle models are a logistic regression and some other GLM combined. You first have a logistic regression which decides if a datapoint going to be zero or not, and then a GLM that decides what the nonzero distribution is. So you end up with two sets of regression coefficients. $\endgroup$
    – John Madden
    Commented Aug 24, 2023 at 4:37

1 Answer 1

Reset to default
1
$\begingroup$

A few thoughts:

First, while in this case the difference in distributions is blatantly obvious, in general, if you want to compare two distributions a QQ plot is much better than two histograms. In one of his books, William S. Cleveland disparages histograms saying "longevity and ubiquity are not guarantees of utility" and never uses them again.

Second, you've already had recommendation for a hurdle model. That's a good idea.

Third, you clearly have both zero inflation and over-dispersion and it's pretty easy to figure out why you have both. Zero inflation because a lot of posts don't get any likes at all. Heck, a huge number of social media posts don't get any views, or get very few. Over-dispersion -- well, almost every set of count data I've seen has this. In context, though, anyone who looks at social media knows that some posts get a TON of likes. E.g. posts by celebrities.

Finally, an alternate to hurdle models is zero-inflated negative binomial models. I did some research comparing ZINB to Hurdle models, but I did it back in grad school, around 25 years ago. Doubtless there is much more now. Maybe someone here is up-to-date on the differences; or you can do some research on which is preferred where. Back then, I found the main difference was by field of research. IIRC, psychology preferred ZINB and economics preferred hurdle, probably because James Heckman, who did a lot of work on hurdle models, is an economist.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thanks for the suggestions, I'll look into both. ZINB seem relatively easy to implement in R according to UCLA. Will also check what the differences are between the two asides from preferences from different research disciplines. $\endgroup$
    – Simone
    Commented Aug 23, 2023 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.