Appropriate GLM when response variable is proportion, but not binomial

The response variable I'm dealing with is the proportion of a total area that is suitable habitat for a species of interest. So although the response variable is bounded between 0 and 1, my intuition is that it wouldn't be appropriate to call it binomial since the numerator and denominator of the proportion are non-integer. The beta distribution comes to mind, but I'm uncertain of the appropriate link function and whether there tools in R to deal with a beta.

Some background on my eventual goal: I'll likely be pursuing a conditional autoregressive model to account for spatial autocorrelation. I'll be treating space as 1-dimensional since I'm dealing with a river system, and so each observational unit only has two neighbors: one upstream and downstream.

I'll be working in R and JAGS/BUGS if I decide to go Bayesian.

• There are a variety of possible distributional models for compositional data. When considering two components, or when focusing on one of the components, some people use beta regression, for example. (With multiple components, there's Dirichlet models.) If you want to use a standard GLM package, you might consider a quasi-binomial, but it will restrict you to a particular form of relationship between variance and mean. A search on compositional data here should turn up a number of questions. May 15 '14 at 19:46

Before venturing into the territory of GLMs it might be worth fitting a regression model on an appropriately transformed version of the response variable. If we let $$0 be the area-proportion (and assuming you don't have any proportions that are exactly zero or one) then a reasonable regression model would be:

$$\log \bigg( \frac{Y_i}{1-Y_i} \bigg) = \beta_0 + \sum_{k=1}^m \beta_1 x_{i,k} + \varepsilon_i \quad \quad \quad \quad \quad \varepsilon_i \sim \text{IID N}(0, \sigma^2).$$

This is a transformation that is closely related to a scaled variant of the hyperbolic tangent function. If we let $$\mu_i \equiv \beta_0 + \sum_k \beta_1 x_{i,k}$$ denote the regression part of the equation then we have:

$$\log \bigg( \frac{Y_i}{1-Y_i} \bigg) = \mu_i + \varepsilon_i \quad \quad \iff \quad \quad Y_i = \frac{\exp(\mu_i+\varepsilon_i)}{1 + \exp(\mu_i+\varepsilon_i)}$$

Obviously this regression equation might not fit your data, particularly if there is complicated spatial autocorrelation. Nevertheless, it is a reasonable starting point for modelling to get a simple understanding of the relationship with expalantory variables. This is a linear regression model that can be fit using standard MLE methods. You can then use diagnostic plots to look for nonlinearity, which would indicate a failure of the transformation. You can also use diagnostic methods to test for spatial auto-correlation, etc., to see if you need to generalise your model.

Beta-regression comes to mind, as you mentioned. Look around the site and the respective tag, .

R has a package, betareg as well.