Appropriate GLM when response variable is proportion, but not binomial The response variable I'm dealing with is the proportion of a total area that is suitable habitat for a species of interest. So although the response variable is bounded between 0 and 1, my intuition is that it wouldn't be appropriate to call it binomial since the numerator and denominator of the proportion are non-integer. The beta distribution comes to mind, but I'm uncertain of the appropriate link function and whether there tools in R to deal with a beta.  
Some background on my eventual goal: I'll likely be pursuing a conditional autoregressive model to account for spatial autocorrelation. I'll be treating space as 1-dimensional since I'm dealing with a river system, and so each observational unit only has two neighbors: one upstream and downstream.
I'll be working in R and JAGS/BUGS if I decide to go Bayesian.
 A: Before venturing into the territory of GLMs it might be worth fitting a regression model on an appropriately transformed version of the response variable.  If we let $0<Y_i<1$ be the area-proportion (and assuming you don't have any proportions that are exactly zero or one) then a reasonable regression model would be:
$$\log \bigg( \frac{Y_i}{1-Y_i} \bigg) = \beta_0 + \sum_{k=1}^m \beta_1 x_{i,k} + \varepsilon_i 
\quad \quad \quad \quad \quad 
\varepsilon_i \sim \text{IID N}(0, \sigma^2).$$
This is a transformation that is closely related to a scaled variant of the hyperbolic tangent function.  If we let $\mu_i \equiv \beta_0 + \sum_k \beta_1 x_{i,k}$ denote the regression part of the equation then we have:
$$\log \bigg( \frac{Y_i}{1-Y_i} \bigg) = \mu_i + \varepsilon_i
\quad \quad \iff \quad \quad 
Y_i = \frac{\exp(\mu_i+\varepsilon_i)}{1 + \exp(\mu_i+\varepsilon_i)}$$
Obviously this regression equation might not fit your data, particularly if there is complicated spatial autocorrelation.  Nevertheless, it is a reasonable starting point for modelling to get a simple understanding of the relationship with expalantory variables.  This is a linear regression model that can be fit using standard MLE methods.  You can then use diagnostic plots to look for nonlinearity, which would indicate a failure of the transformation.  You can also use diagnostic methods to test for spatial auto-correlation, etc., to see if you need to generalise your model.
A: Beta-regression comes to mind, as you mentioned. Look around the site and the respective tag, beta-regression.
R has a package, betareg as well.
