Skip to main content
Added ref. on ZIP
Source Link
Scortchi
  • 31.6k
  • 9
  • 102
  • 282

Consider $n$ counts having the same covariate pattern (replicates). Suppose the $i$th count $X_i$ were to follow a Poisson distribution with its own mean $\mu_i$; the joint mass function of $n$ independent counts is then

$$\begin{align} f_{\vec{X}}(\vec{x};\mu_1,\ldots,\mu_n) &= \prod_{i=1}^n\frac{\mu_i^{x_i}\exp(-\mu_i)}{x_i!}\\ &= \prod_{i=1}^n \frac{1}{x_i!}\cdot\prod_{i=1}^n \mu_i^{x_i}\cdot \exp\left(-\sum_{i=1}^n\mu_i\right) \end{align}$$

Reparametrize with $\phi=\sum_{i=1}^n \mu_i$ & $\pi_i=\frac{\mu_i}{\sum_{i=1}^n \mu_i}$, & let $t=\sum_{i=1}^n x_i$:

$$\begin{align} f_{\vec{X},T}(\vec{x},t;\pi_1,\ldots,\pi_n,\phi)&=\prod_{i=1}^n\frac{1}{x_i!}\cdot\prod_{i=1}^n\pi_i^{x_i}\cdot\phi^{t}\exp(-\phi) \end{align}$$

The statistic $t$ is sufficient for $\phi$ (& ancillary for $\pi_1,\ldots,\pi_n$), with a marginal Poisson distribution having the mass function

$$f_T(t;\phi)= \frac{\phi^t\exp(-\phi)}{t!}$$

So the conditional distribution of $X$ given $T=t$ is multinomial, free of the nuisance parameter $\phi$, having the mass function

$$ f_{X|T}(x;t,\pi_1,\ldots,\pi_n)=\frac{f_{\vec{X},T}(\vec{x},t;\pi_,\ldots,\pi_n,\phi)}{f_{T}(t;\phi)}={\prod_{i=1}^n \frac{t!}{x_i!}}\cdot\prod_{i=1}^n\pi_i^{x_i} $$

With the null hypothesis $\pi_1=\ldots=\pi_n=\frac{1}{n}$, the log likelihood ratio test statistic is

$$2\sum_{i=1}^n X_i \log \frac{X_i}{t/n}$$

& Rao's score test statistic is

$$\sum_{i=1}^n \frac{(X_i-t/n)^2}{t/n}$$

Under the null, both follow, asymptotically, a chi-squared distribution with $n-1$ degrees of freedom ; the resulting tests of goodness of fit are known as the G-test (I don't know why) & Pearson's chi-squared test respectively. Exact distributions of either statistic under the null can be of course be obtained by exhaustive calculation or simulation.

The extension to an omnibus test across $m$ covariate patterns is straightforward, & gives as test statistics

$$2\sum_{j=1}^m\sum_{i=1}^{n_j} X_{ij} \log \frac{X_{ij}}{t_j / n_j}$$

&

$$\sum_{j=1}^m \sum_{i=1}^{n_j} \frac{(X_{ij}-t_j/n_j)^2}{t_j /n_j}$$

both, under the null, asymptotically, following a chi-squared distribution with $\sum_{j=1}^m (n_j-1)$ degrees of freedom.

The score test is, I think, the most popular—the ratio of the sample variance to the sample mean has an intuitive appeal as a measure of extra-Poisson dispersion. Low values may be taken as evidence for under-dispersion.

A variance of counts exceeding their mean may also indicate zero inflation. See Van den Broeck (1995), Biometrics, 51, pp 738–743, "A Score Test for Zero Inflation in a Poisson Distribution" & How to test for Zero-Inflation in a dataset?.

Consider $n$ counts having the same covariate pattern (replicates). Suppose the $i$th count $X_i$ were to follow a Poisson distribution with its own mean $\mu_i$; the joint mass function of $n$ independent counts is then

$$\begin{align} f_{\vec{X}}(\vec{x};\mu_1,\ldots,\mu_n) &= \prod_{i=1}^n\frac{\mu_i^{x_i}\exp(-\mu_i)}{x_i!}\\ &= \prod_{i=1}^n \frac{1}{x_i!}\cdot\prod_{i=1}^n \mu_i^{x_i}\cdot \exp\left(-\sum_{i=1}^n\mu_i\right) \end{align}$$

Reparametrize with $\phi=\sum_{i=1}^n \mu_i$ & $\pi_i=\frac{\mu_i}{\sum_{i=1}^n \mu_i}$, & let $t=\sum_{i=1}^n x_i$:

$$\begin{align} f_{\vec{X},T}(\vec{x},t;\pi_1,\ldots,\pi_n,\phi)&=\prod_{i=1}^n\frac{1}{x_i!}\cdot\prod_{i=1}^n\pi_i^{x_i}\cdot\phi^{t}\exp(-\phi) \end{align}$$

The statistic $t$ is sufficient for $\phi$ (& ancillary for $\pi_1,\ldots,\pi_n$), with a marginal Poisson distribution having the mass function

$$f_T(t;\phi)= \frac{\phi^t\exp(-\phi)}{t!}$$

So the conditional distribution of $X$ given $T=t$ is multinomial, free of the nuisance parameter $\phi$, having the mass function

$$ f_{X|T}(x;t,\pi_1,\ldots,\pi_n)=\frac{f_{\vec{X},T}(\vec{x},t;\pi_,\ldots,\pi_n,\phi)}{f_{T}(t;\phi)}={\prod_{i=1}^n \frac{t!}{x_i!}}\cdot\prod_{i=1}^n\pi_i^{x_i} $$

With the null hypothesis $\pi_1=\ldots=\pi_n=\frac{1}{n}$, the log likelihood ratio test statistic is

$$2\sum_{i=1}^n X_i \log \frac{X_i}{t/n}$$

& Rao's score test statistic is

$$\sum_{i=1}^n \frac{(X_i-t/n)^2}{t/n}$$

Under the null, both follow, asymptotically, a chi-squared distribution with $n-1$ degrees of freedom ; the resulting tests of goodness of fit are known as the G-test (I don't know why) & Pearson's chi-squared test respectively. Exact distributions of either statistic under the null can be of course be obtained by exhaustive calculation or simulation.

The extension to an omnibus test across $m$ covariate patterns is straightforward, & gives as test statistics

$$2\sum_{j=1}^m\sum_{i=1}^{n_j} X_{ij} \log \frac{X_{ij}}{t_j / n_j}$$

&

$$\sum_{j=1}^m \sum_{i=1}^{n_j} \frac{(X_{ij}-t_j/n_j)^2}{t_j /n_j}$$

both, under the null, asymptotically, following a chi-squared distribution with $\sum_{j=1}^m (n_j-1)$ degrees of freedom.

The score test is, I think, the most popular—the ratio of the sample variance to the sample mean has an intuitive appeal as a measure of extra-Poisson dispersion. Low values may be taken as evidence for under-dispersion.

Consider $n$ counts having the same covariate pattern (replicates). Suppose the $i$th count $X_i$ were to follow a Poisson distribution with its own mean $\mu_i$; the joint mass function of $n$ independent counts is then

$$\begin{align} f_{\vec{X}}(\vec{x};\mu_1,\ldots,\mu_n) &= \prod_{i=1}^n\frac{\mu_i^{x_i}\exp(-\mu_i)}{x_i!}\\ &= \prod_{i=1}^n \frac{1}{x_i!}\cdot\prod_{i=1}^n \mu_i^{x_i}\cdot \exp\left(-\sum_{i=1}^n\mu_i\right) \end{align}$$

Reparametrize with $\phi=\sum_{i=1}^n \mu_i$ & $\pi_i=\frac{\mu_i}{\sum_{i=1}^n \mu_i}$, & let $t=\sum_{i=1}^n x_i$:

$$\begin{align} f_{\vec{X},T}(\vec{x},t;\pi_1,\ldots,\pi_n,\phi)&=\prod_{i=1}^n\frac{1}{x_i!}\cdot\prod_{i=1}^n\pi_i^{x_i}\cdot\phi^{t}\exp(-\phi) \end{align}$$

The statistic $t$ is sufficient for $\phi$ (& ancillary for $\pi_1,\ldots,\pi_n$), with a marginal Poisson distribution having the mass function

$$f_T(t;\phi)= \frac{\phi^t\exp(-\phi)}{t!}$$

So the conditional distribution of $X$ given $T=t$ is multinomial, free of the nuisance parameter $\phi$, having the mass function

$$ f_{X|T}(x;t,\pi_1,\ldots,\pi_n)=\frac{f_{\vec{X},T}(\vec{x},t;\pi_,\ldots,\pi_n,\phi)}{f_{T}(t;\phi)}={\prod_{i=1}^n \frac{t!}{x_i!}}\cdot\prod_{i=1}^n\pi_i^{x_i} $$

With the null hypothesis $\pi_1=\ldots=\pi_n=\frac{1}{n}$, the log likelihood ratio test statistic is

$$2\sum_{i=1}^n X_i \log \frac{X_i}{t/n}$$

& Rao's score test statistic is

$$\sum_{i=1}^n \frac{(X_i-t/n)^2}{t/n}$$

Under the null, both follow, asymptotically, a chi-squared distribution with $n-1$ degrees of freedom ; the resulting tests of goodness of fit are known as the G-test (I don't know why) & Pearson's chi-squared test respectively. Exact distributions of either statistic under the null can be of course be obtained by exhaustive calculation or simulation.

The extension to an omnibus test across $m$ covariate patterns is straightforward, & gives as test statistics

$$2\sum_{j=1}^m\sum_{i=1}^{n_j} X_{ij} \log \frac{X_{ij}}{t_j / n_j}$$

&

$$\sum_{j=1}^m \sum_{i=1}^{n_j} \frac{(X_{ij}-t_j/n_j)^2}{t_j /n_j}$$

both, under the null, asymptotically, following a chi-squared distribution with $\sum_{j=1}^m (n_j-1)$ degrees of freedom.

The score test is, I think, the most popular—the ratio of the sample variance to the sample mean has an intuitive appeal as a measure of extra-Poisson dispersion. Low values may be taken as evidence for under-dispersion.

A variance of counts exceeding their mean may also indicate zero inflation. See Van den Broeck (1995), Biometrics, 51, pp 738–743, "A Score Test for Zero Inflation in a Poisson Distribution" & How to test for Zero-Inflation in a dataset?.

added 694 characters in body
Source Link
Scortchi
  • 31.6k
  • 9
  • 102
  • 282

Consider $n$ counts having the same covariate pattern (replicates). Suppose the $i$th count $X_i$ were to follow a Poisson distribution with its own mean $\mu_i$; the joint mass function of $n$ independent counts is then

$$\begin{align} f_{\vec{X}}(\vec{x};\mu_1,\ldots,\mu_n) &= \prod_{i=1}^n\frac{\mu_i^{x_i}\exp(-\mu_i)}{x_i!}\\ &= \prod_{i=1}^n \frac{1}{x_i!}\cdot\prod_{i=1}^n \mu_i^{x_i}\cdot \exp\left(-\sum_{i=1}^n\mu_i\right) \end{align}$$

Reparametrize with $\phi=\sum_{i=1}^n \mu_i$ & $\pi_i=\frac{\mu_i}{\sum_{i=1}^n \mu_i}$, & let $t=\sum_{i=1}^n x_i$:

$$\begin{align} f_{\vec{X},T}(\vec{x},t;\pi_1,\ldots,\pi_n,\phi)&=\prod_{i=1}^n\frac{1}{x_i!}\cdot\prod_{i=1}^n\pi_i^{x_i}\cdot\phi^{t}\exp(-\phi) \end{align}$$

The statistic $t$ is sufficient for $\phi$ (& ancillary for $\pi_1,\ldots,\pi_n$), with a marginal Poisson distribution having the mass function

$$f_T(t;\phi)= \frac{\phi^t\exp(-\phi)}{t!}$$

So the conditional distribution of $X$ given $T=t$ is multinomial, free of the nuisance parameter $\phi$, having the mass function

$$ f_{X|T}(x;t,\pi_1,\ldots,\pi_n)=\frac{f_{\vec{X},T}(\vec{x},t;\pi_,\ldots,\pi_n,\phi)}{f_{T}(t;\phi)}={\prod_{i=1}^n \frac{t!}{x_i!}}\cdot\prod_{i=1}^n\pi_i^{x_i} $$

With the null hypothesis $\pi_1=\ldots=\pi_n=\frac{1}{n}$, the log likelihood ratio test statistic is

$$2\sum_{i=1}^n X_i \log \frac{X_i}{t/n}$$

& Rao's score test statistic is

$$\sum_{i=1}^n \frac{(X_i-t/n)^2}{t/n}$$

Under the null, both follow, asymptotically, a chi-squared distributionsdistribution with $n-1$ degrees of freedom asymptotically;; the resulting tests of goodness of fit are known as the G-test (I don't know why) & Pearson's chi-squared test respectively. Exact distributions of either statistic under the null can be of course be obtained by exhaustive calculation or simulation.

The extension to an omnibus test across $m$ covariate patterns is straightforward, & gives as test statistics

$$2\sum_{j=1}^m\sum_{i=1}^{n_j} X_{ij} \log \frac{X_{ij}}{t_j / n_j}$$

&

$$\sum_{j=1}^m \sum_{i=1}^{n_j} \frac{(X_{ij}-t_j/n_j)^2}{t_j /n_j}$$

both, under the null, asymptotically, following a chi-squared distribution with $\sum_{j=1}^m (n_j-1)$ degrees of freedom.

The score test is, I think, the most popular—the ratio of the sample variance to the sample mean has an intuitive appeal as a measure of extra-Poisson dispersion. Low values may be taken as evidence for under-dispersion.

Suppose the $i$th count $X_i$ were to follow a Poisson distribution with its own mean $\mu_i$; the joint mass function of $n$ independent counts is then

$$\begin{align} f_{\vec{X}}(\vec{x};\mu_1,\ldots,\mu_n) &= \prod_{i=1}^n\frac{\mu_i^{x_i}\exp(-\mu_i)}{x_i!}\\ &= \prod_{i=1}^n \frac{1}{x_i!}\cdot\prod_{i=1}^n \mu_i^{x_i}\cdot \exp\left(-\sum_{i=1}^n\mu_i\right) \end{align}$$

Reparametrize with $\phi=\sum_{i=1}^n \mu_i$ & $\pi_i=\frac{\mu_i}{\sum_{i=1}^n \mu_i}$, & let $t=\sum_{i=1}^n x_i$:

$$\begin{align} f_{\vec{X},T}(\vec{x},t;\pi_1,\ldots,\pi_n,\phi)&=\prod_{i=1}^n\frac{1}{x_i!}\cdot\prod_{i=1}^n\pi_i^{x_i}\cdot\phi^{t}\exp(-\phi) \end{align}$$

The statistic $t$ is sufficient for $\phi$ (& ancillary for $\pi_1,\ldots,\pi_n$), with a marginal Poisson distribution having the mass function

$$f_T(t;\phi)= \frac{\phi^t\exp(-\phi)}{t!}$$

So the conditional distribution of $X$ given $T=t$ is multinomial, free of the nuisance parameter $\phi$, having the mass function

$$ f_{X|T}(x;t,\pi_1,\ldots,\pi_n)=\frac{f_{\vec{X},T}(\vec{x},t;\pi_,\ldots,\pi_n,\phi)}{f_{T}(t;\phi)}={\prod_{i=1}^n \frac{t!}{x_i!}}\cdot\prod_{i=1}^n\pi_i^{x_i} $$

With the null hypothesis $\pi_1=\ldots=\pi_n=\frac{1}{n}$, the log likelihood ratio test statistic is

$$2\sum_{i=1}^n X_i \log \frac{X_i}{t/n}$$

& Rao's score test statistic is

$$\sum_{i=1}^n \frac{(X_i-t/n)^2}{t/n}$$

Under the null both follow chi-squared distributions with $n-1$ degrees of freedom asymptotically; the resulting tests of goodness of fit are known as the G-test (I don't know why) & Pearson's chi-squared test respectively. Exact distributions of either statistic under the null can be of course be obtained by exhaustive calculation or simulation.

Consider $n$ counts having the same covariate pattern (replicates). Suppose the $i$th count $X_i$ were to follow a Poisson distribution with its own mean $\mu_i$; the joint mass function of $n$ independent counts is then

$$\begin{align} f_{\vec{X}}(\vec{x};\mu_1,\ldots,\mu_n) &= \prod_{i=1}^n\frac{\mu_i^{x_i}\exp(-\mu_i)}{x_i!}\\ &= \prod_{i=1}^n \frac{1}{x_i!}\cdot\prod_{i=1}^n \mu_i^{x_i}\cdot \exp\left(-\sum_{i=1}^n\mu_i\right) \end{align}$$

Reparametrize with $\phi=\sum_{i=1}^n \mu_i$ & $\pi_i=\frac{\mu_i}{\sum_{i=1}^n \mu_i}$, & let $t=\sum_{i=1}^n x_i$:

$$\begin{align} f_{\vec{X},T}(\vec{x},t;\pi_1,\ldots,\pi_n,\phi)&=\prod_{i=1}^n\frac{1}{x_i!}\cdot\prod_{i=1}^n\pi_i^{x_i}\cdot\phi^{t}\exp(-\phi) \end{align}$$

The statistic $t$ is sufficient for $\phi$ (& ancillary for $\pi_1,\ldots,\pi_n$), with a marginal Poisson distribution having the mass function

$$f_T(t;\phi)= \frac{\phi^t\exp(-\phi)}{t!}$$

So the conditional distribution of $X$ given $T=t$ is multinomial, free of the nuisance parameter $\phi$, having the mass function

$$ f_{X|T}(x;t,\pi_1,\ldots,\pi_n)=\frac{f_{\vec{X},T}(\vec{x},t;\pi_,\ldots,\pi_n,\phi)}{f_{T}(t;\phi)}={\prod_{i=1}^n \frac{t!}{x_i!}}\cdot\prod_{i=1}^n\pi_i^{x_i} $$

With the null hypothesis $\pi_1=\ldots=\pi_n=\frac{1}{n}$, the log likelihood ratio test statistic is

$$2\sum_{i=1}^n X_i \log \frac{X_i}{t/n}$$

& Rao's score test statistic is

$$\sum_{i=1}^n \frac{(X_i-t/n)^2}{t/n}$$

Under the null, both follow, asymptotically, a chi-squared distribution with $n-1$ degrees of freedom ; the resulting tests of goodness of fit are known as the G-test (I don't know why) & Pearson's chi-squared test respectively. Exact distributions of either statistic under the null can be of course be obtained by exhaustive calculation or simulation.

The extension to an omnibus test across $m$ covariate patterns is straightforward, & gives as test statistics

$$2\sum_{j=1}^m\sum_{i=1}^{n_j} X_{ij} \log \frac{X_{ij}}{t_j / n_j}$$

&

$$\sum_{j=1}^m \sum_{i=1}^{n_j} \frac{(X_{ij}-t_j/n_j)^2}{t_j /n_j}$$

both, under the null, asymptotically, following a chi-squared distribution with $\sum_{j=1}^m (n_j-1)$ degrees of freedom.

The score test is, I think, the most popular—the ratio of the sample variance to the sample mean has an intuitive appeal as a measure of extra-Poisson dispersion. Low values may be taken as evidence for under-dispersion.

Source Link
Scortchi
  • 31.6k
  • 9
  • 102
  • 282

Suppose the $i$th count $X_i$ were to follow a Poisson distribution with its own mean $\mu_i$; the joint mass function of $n$ independent counts is then

$$\begin{align} f_{\vec{X}}(\vec{x};\mu_1,\ldots,\mu_n) &= \prod_{i=1}^n\frac{\mu_i^{x_i}\exp(-\mu_i)}{x_i!}\\ &= \prod_{i=1}^n \frac{1}{x_i!}\cdot\prod_{i=1}^n \mu_i^{x_i}\cdot \exp\left(-\sum_{i=1}^n\mu_i\right) \end{align}$$

Reparametrize with $\phi=\sum_{i=1}^n \mu_i$ & $\pi_i=\frac{\mu_i}{\sum_{i=1}^n \mu_i}$, & let $t=\sum_{i=1}^n x_i$:

$$\begin{align} f_{\vec{X},T}(\vec{x},t;\pi_1,\ldots,\pi_n,\phi)&=\prod_{i=1}^n\frac{1}{x_i!}\cdot\prod_{i=1}^n\pi_i^{x_i}\cdot\phi^{t}\exp(-\phi) \end{align}$$

The statistic $t$ is sufficient for $\phi$ (& ancillary for $\pi_1,\ldots,\pi_n$), with a marginal Poisson distribution having the mass function

$$f_T(t;\phi)= \frac{\phi^t\exp(-\phi)}{t!}$$

So the conditional distribution of $X$ given $T=t$ is multinomial, free of the nuisance parameter $\phi$, having the mass function

$$ f_{X|T}(x;t,\pi_1,\ldots,\pi_n)=\frac{f_{\vec{X},T}(\vec{x},t;\pi_,\ldots,\pi_n,\phi)}{f_{T}(t;\phi)}={\prod_{i=1}^n \frac{t!}{x_i!}}\cdot\prod_{i=1}^n\pi_i^{x_i} $$

With the null hypothesis $\pi_1=\ldots=\pi_n=\frac{1}{n}$, the log likelihood ratio test statistic is

$$2\sum_{i=1}^n X_i \log \frac{X_i}{t/n}$$

& Rao's score test statistic is

$$\sum_{i=1}^n \frac{(X_i-t/n)^2}{t/n}$$

Under the null both follow chi-squared distributions with $n-1$ degrees of freedom asymptotically; the resulting tests of goodness of fit are known as the G-test (I don't know why) & Pearson's chi-squared test respectively. Exact distributions of either statistic under the null can be of course be obtained by exhaustive calculation or simulation.