Low sample size: LR vs F - test

Question

Some of you might have read this nice paper:

O’Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods in Ecology and Evolution 1:118–122. klick.

Currently I am comparing negative binomial models with gaussian models on transformed data. Unlike O’Hara RB, Kotze DJ (2010) I'm looking at the special case of low sample sizes and in a hypothesis testing context.

A used simulations to investigate the differences between both.

Type I Error simulations

All compuations have been done in R.

I simulated data from a factorial design with one control group ($μ_c$) and 5 treatment groups ($μ_{1−5}$). Abundances were drawn from a negative binomial distributions with fixed dispersion parameter (θ=3.91). Abundances were equal in all treatments.

For the simulations I varied the sample size (3, 6, 9, 12) and the abundances (2, 4 ,8, ... , 1024). 100 datasets were generated and analysed using a negative binomial GLM (MASS:::glm.nb()), a quasipoisson GLM (glm(..., family = 'quasipoisson') and a gaussian GLM + log-transformed data (lm(...)).

I compared the models with the null model using a Likelihood-Ratio test (lmtest:::lrtest()) (gaussian GLM and neg. bin GLM) as well as F-tests (gaussian GLM and quasipoisson GLM)(anova(...test = 'F')).

If needed I can provide the R code, but see also here for a related question of mine.

Results

For small sample sizes, the LR-tests (green - neg.bin.; red - gaussian) lead to an increased Type-I error. The F-test (blue - gaussian, purple - quasi-poisson) seem to work even for small sample sizes.

LR tests give similar (increased) Type I errors for both LM and GLM.

Interestingly the quasi-poisson works pretty well (but also with an F-Test).

As expected, if sample size increases LR-Test performs also well (asymptotically correct).

For the small sample size there have been some convergence problems (not show) for the GLM, however only at low abundances, so source of error can be neglected.

Questions

1. Note the data was generated from a neg.bin. model - so I would have expected that the GLM performs best. However in this case a linear model on transformed abundances performs better. Same for quasi-poisson (F-Test). I suspect this is because of the F-test is doing better with small sample sizes - is this correct and why?

2. The LR-Test does not perform well because of asymptotics. Are the possibilities for improvement?

3. Are there other tests for GLMs which may perform better? How can I improve testing for GLMs?

4. What type of models for count data with small sample sizes should be used?

Edit:

Interestingly, the LR-Test for a binomial GLM does work pretty well:

Here i draw data from a binomial distribution, setup similar as above.

Red: gaussian model (LR-Test + arcsin transformation), Ocher: Binomial GLM (LR-Test), Green: gaussian model (F-Test + arcsin transformation), Blue: Quasibinonial GLM (F-test), Purple: Non-parametric.

Here only the gaussian model (LR-Test + arcsin transformation) shows an increase Type I error, whereas the GLM (LR-Test) does pretty well in terms of Type I error. So there seems to be also a difference between distributions (or maybe glm vs. glm.nb?).

Answer 1

The Likelihood ratio test you're using uses a chi-square distribution to approximate the null distribution of likelihoods. This approximation works best with large sample sizes, so its inaccuracy with a small sample size makes some sense.

I see a few options for getting better Type-I error in your situation:

• There are corrected versions of the likelihood ratio test, such as Bartlett's correction. I don't know much about these (beyond the fact that they exist), but I've heard that Ben Bolker knows more.
• You could estimate the null distribution for the likelihood ratio by bootstrapping. If the observed likelihood ratio falls outside middle 95% of the bootstrap distribution, then it's statistically significant.

Finally, the Poisson distribution has one fewer free parameter than the negative binomial, and might be worth trying when the sample size is very small.