I need to model an unusual response variable (at least to the best of my knowledge) that is: similarity (estimated by Morisita-Horn index) in species composition between pairs of sites.
My response variable is a (continuous) similarity matrix with around 6000 site pairs. (Obs.: I am aware that similarity measures of paired sites are not independent of each other. So, I will need to perform permutation tests to obtain reliable estimates. This is not a problem at the moment.)
The values vary from 0 to 0.99. In my dataset, values are strongly right-skewed, with around 1500 site pairs with similarity < 0.001 (e.g. site pairs that share only one species). When log-transformed, it hardly tends to a normal distribution.
As predictors I have four variables: 3 continuous and one categorical (with three categories). The continuous variables are: geographic distance between each site pair (dist), climatic similarity (clim), and a measure of community dispersal ability (disp), also calculated for each site pair. My categorical variable (env_type) describes the forest types (F1 and/or F2) of the two sites in each pair. Thus, I have a categorical variable with three factors (F1F1, F2F2 or F1F2). Besides the main effects of the above mentioned response variables, I also need to test some interactions among some variables.
So, the question is: What is the indicated family of GLM and the respective link function to deal with this similarity index as response variable?
I found only one paper that applied a Gaussian GLM with log link to similarity data [Gomez-Rodrigues & Baselga (2018) https://onlinelibrary.wiley.com/doi/abs/10.1111/ecog.03693], but in their case the similarity matrix seems to be not so right-skewed. In text books I couldn´t find specific advice on similarities as response variable. I think it is important to highlight that I'm not dealing with proportions directly linked to discrete count data. If so, I could use a binomial distribution, but for similarity indices it seems not suitable, because it is intrinsically continuous. Furthermore, my specific dataset is strongly right-skewed and this imposes an additional problem.