0
$\begingroup$

Following this example, I am fitting a glm model with rstanarm to count data that look like this:

enter image description here

The simple specification below runs just fine:

stan_glm(outcome_count ~ 1 + scaled_input + binary_predictor
            , family = neg_binomial_2(link = "log")
            , data = input_data)

But it yields fitted values that are too small and doesn't seem to account very well for the overdispersion in data. Here posterior predictive (yrep) distributions for the max, min, mean and sd of the outcome plotted alongside the values from the raw counts (y):

enter image description here

Using family = poisson(link = "log") yields very similar results. I would have expected the negative binomial to do a much better job of accounting for the overdispersion. Any clues as to what is going on or what I am missing?

$\endgroup$
5
  • 1
    $\begingroup$ Not sure how that output helps. What do the raw data look like, please? Or if you are showing them somehow, please help us to interpret your graphs. $\endgroup$
    – Nick Cox
    Aug 1, 2022 at 17:48
  • $\begingroup$ Fair enough - I have edited the question to be more informative in that regard. $\endgroup$
    – atmo
    Aug 1, 2022 at 18:28
  • $\begingroup$ That helps.. Fact is a negative binomial isn't indefinitely flexible. Evidently you can't match the long right tail of your data. It might help to compare your sample moments and moment-based summaries (including skewness and kurtosis) with what is possible in a negative binomial. $\endgroup$
    – Nick Cox
    Aug 2, 2022 at 8:15
  • $\begingroup$ Right, that makes sense. Any suggestions of directions to look in order to accomodate long tails like this? Otherwise I guess I should think about ways to restate my analytical question... $\endgroup$
    – atmo
    Aug 2, 2022 at 11:47
  • $\begingroup$ Sorry, no more ideas from me. A good suggestion would need a better idea of the variable and how it is produced. $\endgroup$
    – Nick Cox
    Aug 2, 2022 at 13:55

1 Answer 1

0
$\begingroup$

I got the model to work with the following two steps:

  1. Most observations in the long tail was associated with one of the states of my binary predictor. I excluded observations in this state from the model (I might return to modelling these back in at a later point though), and the model ran just fine. It still didn't fit the data well, but the sampling ran just fine.
  2. I introduced a new grouping variable and allowed intercepts to vary according to their grouping using stan_glmer, and put a tight prior on the parameters. The model ran without issues and the introduction of varying intercepts improved the model fit considerably.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.