I want to model mean colony size as a function of predator density. Within each treatment level the smaller colonies (3-5 individuals) are a lot more common than the large ones (up to 600 individuals). Which error distribution should I use?
This is a partial answer only, but the graphical content makes a comment a poor alternative. In a comment, the OP talks about using
sqrt(log()) as a transformation.
I'd advise against that on the general grounds that it is a very unusual and idiosyncratic transformation, so you will face puzzlement at all levels up to supervisors, examiners and paper reviewers. (Translate to the terminology of your own education and research set-up.)
I'd also advise that it really stretches out low colony sizes. Here is a plot for sizes 1(1)600.
Notice how on this transformed scale the interval from size 1 to size 5 is more than half the entire range from size 1 to size 600.
Implication: this transformation will create outliers for small colony sizes. The better fit observed could be an artefact of over-transforming.
I wouldn't go further than a log link. At least that is standard and easier to think about.
There's not enough information to offer a definitive answer.
Colony size would be a count, I gather (rather than say an area, or a mass).
It's better not to think of them as representing an error term but rather the conditional distribution of the response.
Some possible models include negative binomial or log series distribution -- these are two distributions in the exponential-family which could be suitable for such counts, though they're not the only possibilities (some care must betaken with the zero-case; for example the negative binomial used in GLMs starts from 0, but colony sizes are presumably bounded below by 1).
In some cases you may be able to get some use from the Poisson but I expect that typically it won't be nearly skewed enough.
(The log series distribution may not be implemented in most software.)
You may need to consider more complex models that straight exponential family. You might need to consider truncated or shifted distributions, or if zeroes are possible, perhaps zero-inflated or hurdle models; you may also need mixed effect GLMs for example.
About transformation, see :
O’Hara, Robert B., and D. Johan Kotze. 2010. « Do Not Log-Transform Count Data ». Methods in Ecology and Evolution 1 (2): 118‑22. doi:10.1111/j.2041-210X.2010.00021.x.
1. Ecological count data (e.g. number of individuals or species) are often log-transformed to satisfy parametric test assumptions.
2. Apart from the fact that generalized linear models are better suited in dealing with count data, a log-transformation of counts has the additional quandary in how to deal with zero observations. With just one zero observation (if this observation represents a sampling unit), the whole data set needs to be fudged by adding a value (usually 1) before transformation.
3. Simulating data from a negative binomial distribution, we compared the outcome of fitting models that were transformed in various ways (log, square root) with results from fitting models using quasi-Poisson and negative binomial models to untransformed count data.
4. We found that the transformations performed poorly, except when the dispersion was small and the mean counts were large. The quasi-Poisson and negative binomial models consistently performed well, with little bias.
5. We recommend that count data should not be analysed by log-transforming it, but instead models based on Poisson and negative binomial distributions should be used.