# What GLM family and link function to use with similarity index as response variable?

I need to model an unusual response variable (at least to the best of my knowledge) that is: similarity (estimated by Morisita-Horn index) in species composition between pairs of sites.

My response variable is a (continuous) similarity matrix with around 6000 site pairs. (Obs.: I am aware that similarity measures of paired sites are not independent of each other. So, I will need to perform permutation tests to obtain reliable estimates. This is not a problem at the moment.)

The values vary from 0 to 0.99. In my dataset, values are strongly right-skewed, with around 1500 site pairs with similarity < 0.001 (e.g. site pairs that share only one species). When log-transformed, it hardly tends to a normal distribution.

As predictors I have four variables: 3 continuous and one categorical (with three categories). The continuous variables are: geographic distance between each site pair (dist), climatic similarity (clim), and a measure of community dispersal ability (disp), also calculated for each site pair. My categorical variable (env_type) describes the forest types (F1 and/or F2) of the two sites in each pair. Thus, I have a categorical variable with three factors (F1F1, F2F2 or F1F2). Besides the main effects of the above mentioned response variables, I also need to test some interactions among some variables.

So, the question is: What is the indicated family of GLM and the respective link function to deal with this similarity index as response variable?

I found only one paper that applied a Gaussian GLM with log link to similarity data [Gomez-Rodrigues & Baselga (2018) https://onlinelibrary.wiley.com/doi/abs/10.1111/ecog.03693], but in their case the similarity matrix seems to be not so right-skewed. In text books I couldn´t find specific advice on similarities as response variable. I think it is important to highlight that I'm not dealing with proportions directly linked to discrete count data. If so, I could use a binomial distribution, but for similarity indices it seems not suitable, because it is intrinsically continuous. Furthermore, my specific dataset is strongly right-skewed and this imposes an additional problem.

• my comments got lost on migration (not sure this is enough for a full answer), but consider (1) beta regression; (2) plain old linear models with permutation tests to assess significance. Also, note that you're showing us histograms of the marginal distribution of the response, which isn't always the same as the conditional distribution (which is what distributional assumptions refer to). Mar 29, 2019 at 12:44
• If similarity can be zero, as your question states, where are such values shown on your histogram of log index? Apr 1, 2019 at 7:31
• How proportions are measured is not as important as might be guessed. If any measure is bounded on $[0, 1]$ then as the mean approaches 0 or 1, so also the variance must approach 0. You're describing your measure as continuous, but it's also discrete, with many possible values. There is much territory where discrete responses can be treated as approximately continuous. Apr 1, 2019 at 7:38
• @NickCox and Peter Flom. This question doesn't seem to me to be a duplicate, although a cross reference to the previous question is relevant. The current question was specifically about GLM families whereas the previous question was not. The accepted answer to the previous question was to use the betareg package, but that is not a GLM approach and therefore does not answer the current question. Apr 3, 2019 at 23:25
• @NickCox I don't agree that the question already has a correct answer, for the reason I already explained. In addition, I note that OP has gone to a lot of trouble to write a complete question with a lot of background detail that is unique to his question and there is considerable scope for addressing these details in an answer should someone choose to do so. IMO one should be very careful about closing a question like this, especially when the earlier question was so short and vague. In the earlier question we don't even know whether the data is continuous or not, as gung noted in his answer. Apr 5, 2019 at 23:20

There is no generalized linear model (GLM) for continuous proportion data on (0,1), but there are two approximate possibilities.

The first possibility would be a quasi-GLM family with a beta distribution type of variance function: $$V(\mu)=\mu^\alpha(1-\mu)^\beta.$$ You would however have to estimate the variance parameters $$\alpha$$ and $$\beta$$ or set them to somewhat arbitrary values, like $$\alpha=\beta=1$$ or $$\alpha=\beta=0.5$$.

For example, you might use a quasi-binomial GLM family in R (by setting family=quasibinomial()) and that would be equivalent to the above variance function with $$\alpha=\beta=1$$. The quasi-family does not make any assumptions about whether the response is discrete or continuous. The logit link would be appropriate, and that's the default for the quasi-binomial family.

Note it is very important that you use a quasi-binomial model rather than an ordinary binomial model because the former allows a dispersion parameter to be estimated from the data.

The second possibility would be to use a gamma GLM with log-link. Although your response variable is constrained to be $$\le 1$$, instead of unbounded as the gamma distribution would imply, the gamma model will nevertheless work quite well for your data because the majority of your responses are less than 0.1 with the upper bound not coming much into play.

• Generalised linear models with logit link and binomial family also may work well, as long as your software offers a handle to calculate standard errors reasonably. stata-journal.com/sjpdf.html?articlenum=st0147 has Stata flavour but people with different favourite software should find ways to do it (if not, you need to change your favourite software). Wedderburn's 1974 paper in Biometrika remains pertinent: jstor.org/stable/2334725 (although the abstract doesn't make it obvious). Apr 1, 2019 at 7:29
• @NickCox A quasi-binomial GLM is a way of implementing my first suggestion with $\alpha=\beta=1$, I wouldn't recommend a straight binomial family as one has to allow for overdispersion. The example in Wedderburn's 1974 paper is the same as my first suggestion with $\alpha=\beta=2$. Indeed I had Wedderburn's paper in mind when I wrote my answer. Apr 1, 2019 at 11:33
• Similarly, I had the Stata implementation of glm in mind and its ability to take "binomial" less than literally. So, my suggestion doesn't commit the user, implicitly or explicitly, to a distribution assumption that may be met poorly. Apr 1, 2019 at 11:49
• Thanks a lot for your reply @Gordon Smyth! As I said on comments above, I´m tempted to use beta regression. You said quasi-GLM family is an "aproximated" possibility. Your aswer conducted me to this question. There, I also found beta regression is an apropriate approach. Do you see any impediment on that? Apr 10, 2019 at 14:43