47
votes
Accepted
Is a "hurdle model" really one model? Or just two separate, sequential models?
Separating the log-likelihood
It is correct that most hurdle models can be estimated separately (I would say, instead of sequentially). The reason is that the log-likelihood can be decomposed into ...
- 15.2k
31
votes
Accepted
Diagnostics for generalized linear (mixed) models (specifically residuals)
This answer is not based on my knowledge but rather quotes what Bolker et al. (2009) wrote in an influential paper in the journal Trends in Ecology and Evolution. Since the article is not open access (...
- 5,556
30
votes
Accepted
Continuous generalization of the negative binomial distribution
That's an interesting question. My research group has been using the distribution you refer to for some years in our publicly available bioinformatics software. As far as I know, the distribution does ...
- 12.1k
27
votes
Accepted
Identical coefficients estimated in Poisson vs Quasi-Poisson model
This is almost a duplicate; the linked question explains that you shouldn't expect the coefficient estimates, residual deviance, nor degrees of freedom to change. The only thing that changes when ...
- 41.4k
22
votes
What regression model is the most appropriate to use with count data?
No, there is no general count data regression model.
(Just as there is no general regression model for continuous data. A linear model with normally distributed homoskedastic noise is most commonly ...
- 117k
19
votes
Continuous generalization of the negative binomial distribution
Look at this paper: Chandra, Nimai Kumar, and Dilip Roy. A continuous version of the negative binomial distribution. Statistica 72, no. 1 (2012): 81.
It's defined in the paper as the survival ...
- 60k
18
votes
Quantifying diversity of bird species
Just about any general book on ecological methods has a section on diversity measures and there are indeed several dedicated monographs on diversity in ecology alone, to say nothing about related ...
- 53.9k
17
votes
Diagnostics for generalized linear (mixed) models (specifically residuals)
This is an old question, but I thought it would be useful to add that option 4 suggested by the OP is now available in the DHARMa R package (available from CRAN, see here).
The package makes the ...
- 7,729
17
votes
How to correctly compute mutual information (Python Example)?
To calculate mutual information, you need to know the distribution of the pair $(X,Y)$ which is counts for each possible value of the pair. This would be described by a 2 dimensional matrix as in ...
- 7,967
15
votes
Clustering of very skewed, count data: any suggestions to go about (transform etc)?
It is not wise to transform the variables individually because they belong together (as you noticed) and to do k-means because the data are counts (you might, but k-means is better to do on continuous ...
- 56.5k
15
votes
Accepted
How to formulate the offset of a GLM
I don't know where you heard that a Poisson or negative binomial with an offset is preferable to a binomial model for a number of individuals surviving out of an initial number; I would normally ...
- 41.4k
14
votes
Accepted
Correlation between continuous data and count data
I'd say there are at least 3 decent options that would make sense for you:
Polyserial Correlation - This would be the most exotic of the 3 options and involves an approximation of a latent, ...
- 810
14
votes
Relationship between Poisson, binomial, negative binomial distributions and normal distribution
The binomial distribution is the distribution of the number of successes in a fixed (i.e. not random) number of independent trials with the same probability of success on each trial. It support is ...
- 8,796
12
votes
Robust estimators for count data
Yes, there are. To name just one, which I've had good experiences with, you can minimize the Cramer-von Mises distance between the empirical distribution and the theoretical distribution with ...
- 37.6k
11
votes
Clustering of very skewed, count data: any suggestions to go about (transform etc)?
@ttnphns has provided a good answer.
Doing clustering well is often about thinking very hard about your data, so let's do some of that. To my mind, the most fundamental aspect of your data is that ...
11
votes
Accepted
Forecasting Poisson, accuracy and prediction intervals
You have what is called intermittent demand, that is, a demand time series characterized by "many" zeros. (If your time series is not demand per se, most of what follows will still apply.) So a web ...
- 117k
11
votes
What regression model is the most appropriate to use with count data?
The "default", the most commonly used and described, distribution of choice for count data is the Poisson distribution. Most often it is illustrated using example of its first practical usage:
A ...
Tim♦
- 135k
10
votes
Robust estimators for count data
If the issue merely boils down to very high or very low observations, one would be tempted to just use a trimmed mean. The problem with that of course, is that your estimate may be biased. You could ...
- 60.4k
9
votes
Estimating probability of attack in Ukraine, given count data
This is not an answer, but rather a side comment:
Keep in mind that the new attacks are not independent of the previous ones. Historical data is not necessarily relevant for the future. It is probably ...
Tim♦
- 135k
9
votes
Quantifying diversity of bird species
Firstly, I assume that you are talking about alpha-diversity (although the existence of multiple sites where the data were taken suggests that beta-diversity could be also relevant).
The simplest ...
- 3,590
8
votes
Quantifying diversity of bird species
The other thing that's worth taking into account (although it involves similar deep rabbit holes as the other aspects of diversity metrics mentioned in other answers and comments) is that diversity ...
- 41.4k
8
votes
Accepted
Poisson regression with small denominators/counts
First of all, a binomial GLM is more appropriate than a Poisson GLM. (A Poisson GLM is used for unbounded counts; your counts are bounded by the total number of surgeries.) The counts aren't that ...
- 923
7
votes
Accepted
Appropriate application of Poisson regression?
A couple of thoughts that may or may not help below. It's a bit hard for us to be helpful without seeing your actual data...
First of all, your data rather obviously does not follow a standard ...
- 117k
7
votes
How to do an ANOVA when your data are non-normal with possibly differing variances?
I notice that your response is called "Number of Organisms", and that all the values are non-negative integers. I suspect these are count data. They should not be treated as normally ...
7
votes
How to do an ANOVA when your data are non-normal with possibly differing variances?
The data, as indicated by the variable names in the linked-to spreadsheet, pertains to number of organisms in 13 groups, so are some kind of count data. We could do better here if we knew some more ...
- 74.6k
7
votes
Dealing with outliers in dependent variables?
The number of visitors is a counted variable and I would expect it to be highly skewed. A first model to try might be Poisson regression, which is equivalent to working on a log scale (specifically, ...
- 53.9k
7
votes
Accepted
Why GLM Poisson model predict negative value for count data?
The Poisson GLM fits a model $y_i \sim \text{Pois}(\mu_i)$ with $\log(\mu_i) = x_i^\top \beta$, i.e., a log links the expectation $\mu_i$ to the so-called "linear predictor" $x_i^\top \beta$, often ...
- 15.2k
7
votes
Accepted
Confidence intervals for the mean of a sample of counts
It's a bit nuanced.
You could pull out the big guns and use a poisson regression
...
- 33.4k
7
votes
Pair-matched count regression in R with offset?
You should be able to do this with a mixed model with a count response (e.g. Poisson or negative binomial): you want the "standard" count-GLM-with-offset model with random variation in the ...
- 41.4k
7
votes
Estimating probability of attack in Ukraine, given count data
Does anyone know what kind of model I would use for something like this?
...
I was just wondering if anyone know some common approaches.
Two approaches you may want to look into:
"Self exciting ...
- 2,160
Only top scored, non community-wiki answers of a minimum length are eligible